Nov 02 19:49:07.343217 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 19:49:07.343228 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 19:49:07.343237 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 19:49:07.343571 ip-10-0-133-26 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 19:49:17.592406 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 19:49:17.592422 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dbf57f0fa209470a89a649c1b920b0f7 --
Nov 02 19:51:06.965139 ip-10-0-133-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:51:07.333586 ip-10-0-133-26 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:07.333586 ip-10-0-133-26 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:51:07.333586 ip-10-0-133-26 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:07.333586 ip-10-0-133-26 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:07.338918 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338834    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.338918 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338916    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.338918 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338920    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338924    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338927    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338930    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338933    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338937    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338941    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338944    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338947    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338949    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338952    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338955    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338965    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338969    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338971    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338974    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338977    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338979    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338982    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.339017 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338984    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338987    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338990    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338993    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338995    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.338998    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339001    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339003    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339006    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339009    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339011    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339014    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339016    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339019    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339022    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339025    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339029    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339034    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339038    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.339456 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339041    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339044    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339046    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339049    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339052    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339055    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339058    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339060    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339063    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339065    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339068    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339071    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339073    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339076    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339079    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339084    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339087    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339090    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339093    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339095    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.339928 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339098    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339101    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339103    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339106    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339108    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339112    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339115    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339117    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339120    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339122    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339125    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339128    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339131    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339134    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339137    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339140    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339142    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339145    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339148    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.340397 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339150    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339153    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339155    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339158    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339161    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339164    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339167    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339170    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339173    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339176    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339179    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339181    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339187    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339189    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339192    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339194    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339197    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339199    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339202    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.340865 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339204    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339207    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339209    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339212    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339214    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339217    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.339221    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340015    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340023    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340026    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340029    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340032    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340035    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340038    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340041    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340044    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340046    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340049    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340052    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.341347 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340055    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340058    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340061    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340064    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340067    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340070    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340073    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340076    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340078    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340081    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340084    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340086    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340089    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340092    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340096    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340100    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340103    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340107    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340110    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.341850 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340112    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340116    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340119    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340123    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340125    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340128    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340131    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340133    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340136    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340138    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340143    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340148    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340152    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340155    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340159    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340162    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340164    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340167    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340170    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.342317 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340173    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340176    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340178    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340181    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340184    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340186    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340189    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340192    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340194    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340197    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340199    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340202    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340205    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340208    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340210    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340213    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340216    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340219    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340222    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340225    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.342880 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340227    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340230    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340232    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340235    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340238    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340240    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340243    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340246    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340250    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340252    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340255    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340257    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340260    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340262    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340265    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340268    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340270    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340273    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340275    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340278    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.343357 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340280    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340283    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340285    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340289    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340292    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340295    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340297    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340300    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340303    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340306    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340308    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340311    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340314    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340316    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340319    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340322    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340802    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340808    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340811    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.343857 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340814    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340816    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340820    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340822    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340825    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340827    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340830    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340833    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340835    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340838    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340840    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340843    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340846    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340849    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340851    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340854    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340857    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340860    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340863    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340865    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.344313 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340868    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340871    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340874    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340877    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340880    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340883    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340886    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340889    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340891    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340895    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340898    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340901    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340903    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340906    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340908    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340911    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340913    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340916    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340919    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.344776 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340922    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340924    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340927    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340929    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340932    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340934    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340937    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340940    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340943    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340946    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340950    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340954    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340957    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340960    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340962    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340966    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340969    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340972    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340975    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.345251 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340978    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340981    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340983    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340986    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340989    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340991    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340994    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340997    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.340999    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341002    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341004    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341007    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341009    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341012    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341014    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341017    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341019    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341022    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341024    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.345697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341027    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341029    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341032    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341035    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341037    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341040    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341042    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341045    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341048    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341050    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341053    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341056    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341059    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341061    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341064    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341067    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341072    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341075    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341078    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.346157 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341080    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341082    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341085    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341088    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341090    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.341093    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342146    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342156    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342164    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342169    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342174    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342177    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342186    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342190    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342193    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342197    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342200    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342203    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342207    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342210    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342214    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342217    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342219    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342223    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:51:07.346676 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342227    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342230    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342234    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342237    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342240    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342244    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342247    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342251    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342254    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342257    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342260    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342263    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342266    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342275    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342281    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342284    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342287    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342290    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342293    2579 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342296    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342302    2579 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342305    2579 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342308    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342312    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:51:07.347305 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342314    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342319    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342322    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342325    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342328    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342332    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342334    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342338    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342342    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342345    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342348    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342351    2579 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342356    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342359    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342362    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342365    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342368    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342371    2579 flags.go:64] FLAG: --help="false"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342374    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342378    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342381    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342384    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342387    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:51:07.347945 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342390    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342394    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342397    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342400    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342403    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342407    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342411    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342414    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342417    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342420    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342423    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342426    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342429    2579 flags.go:64] FLAG: --lock-file=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342432    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342435    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342438    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342443    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342446    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342450    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342453    2579 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342456    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342459    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342463    2579 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342466    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342471    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:51:07.349265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342474    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342478    2579 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342481    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342484    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342487    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342490    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342493    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342496    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342499    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342507    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342510    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342515    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342535    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342540    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342543    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342547    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342550    2579 flags.go:64] FLAG: --port="10250"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342553    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342556    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0820451ca3f24c196"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342559    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342562    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342566    2579 flags.go:64] FLAG: --register-node="true"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342569    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342573    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342576    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342579    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:51:07.349885 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342583    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342587    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342591    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342594    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342597    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342600    2579 flags.go:64] FLAG: --runonce="false"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342603    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342606    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342609    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342612    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342615    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342618    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342622    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342624    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342628    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342630    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342633    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342637    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342641    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342644    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342647    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342652    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342655    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342658    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342662    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:51:07.350839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342665    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342668    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342671    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342674    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342677    2579 flags.go:64] FLAG: --v="2"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342681    2579 flags.go:64] FLAG: --version="false"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342686    2579 flags.go:64] FLAG: --vmodule=""
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342690    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.342694    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342844    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342848    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342851    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342854    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342857    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342860    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342863    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342865    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342868    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342871    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342874    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342876    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.351817 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342879    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342882    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342885    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342889    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342892    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342896    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342898    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342901    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342904    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342907    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342909    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342912    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342915    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342918    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342923    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342925    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342928    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342931    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342934    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.352360 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342937    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342940    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342942    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342945    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342947    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342950    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342953    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342955    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342958    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342960    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342963    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342965    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342968    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342971    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342973    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342978    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342982    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342984    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342987    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342990    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.352856 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342993    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342995    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.342998    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343001    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343003    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343006    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343008    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343011    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343013    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343016    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343019    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343021    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343024    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343027    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343030    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343033    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343036    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343038    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343041    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.353339 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343043    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343046    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343049    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343051    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343054    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343057    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343059    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343065    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343071    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343075    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343078    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343081    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343083    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343086    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343089    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343091    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343094    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343097    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343099    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.353812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343103    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343105    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343108    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343111    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343113    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343116    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343118    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343121    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343126    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343128    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343132    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343135    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343138    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343141    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343144    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.343170    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:07.354298 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343206    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343210    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343213    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343216    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343219    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343224    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343227    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343230    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343233    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343236    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343239    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343241    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343244    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343247    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343250    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343254    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343257    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343260    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343263    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.354686 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343266    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343268    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343271    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343273    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343276    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343279    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343282    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343285    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343287    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343290    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343292    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343295    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343297    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343300    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343302    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343305    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343309    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343312    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343317    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343320    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.355222 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343323    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343325    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343328    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343331    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343334    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343336    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343339    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343342    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343344    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343347    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343349    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343352    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343357    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343360    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343363    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343366    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343369    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343373    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343375    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.355701 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343378    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343381    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343383    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343386    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343389    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343391    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343394    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343396    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343399    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343401    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343406    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343409    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343412    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343414    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343417    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343420    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343422    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343425    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343427    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343430    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.356197 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343433    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343435    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343438    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343440    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343443    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343445    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343448    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343451    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343453    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343456    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343458    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343461    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343464    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343466    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343469    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343471    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343474    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343477    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343479    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.356707 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343482    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343484    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343487    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343492    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343494    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343497    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343499    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.343502    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.351812    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.351828    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.352535    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.352541    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352616    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352621    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352626    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352630    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352636    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352639    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352643    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.357215 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352646    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352649    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352652    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352655    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352658    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352661    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352664    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352667    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352670    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352673    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352675    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352678    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352681    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352684    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352688    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352690    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352693    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352696    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352699    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352701    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.357830 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352704    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352707    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352710    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352713    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352716    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352718    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352721    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352724    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352727    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352729    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352732    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352734    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352737    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352740    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352742    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352745    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352747    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352750    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352753    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352755    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.358323 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352758    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352760    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352763    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352766    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352770    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352774    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352777    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352795    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352801    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352805    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352809    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352812    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352815    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352819    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352822    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352825    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352828    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352831    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352833    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.358947 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352836    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352839    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352842    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352844    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352847    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352850    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352853    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352856    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352859    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352862    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352868    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352870    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352875    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352877    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352880    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352883    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352886    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352888    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352892    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.359398 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352894    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352897    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352899    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352902    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352905    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352908    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352911    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352915    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352917    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352920    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352923    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352926    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352928    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352931    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352934    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352937    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352939    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352942    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352945    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.360040 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.352947    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.352977    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353124    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353129    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353133    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353135    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353139    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353142    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353145    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353148    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353153    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353156    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353159    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353161    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.360807 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353164    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353167    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353170    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353172    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353175    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353179    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353182    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353185    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353187    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353190    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353193    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353195    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353198    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353201    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353204    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353207    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353209    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353212    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353215    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353218    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.361181 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353220    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353223    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353226    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353229    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353231    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353234    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353237    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353240    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353242    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353245    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353247    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353250    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353253    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353256    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353259    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353261    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353264    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353266    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353269    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353272    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.361742 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353274    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353277    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353280    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353282    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353285    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353288    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353290    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353293    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353295    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353298    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353300    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353303    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353305    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353308    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353311    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353314    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353316    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353319    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353322    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353324    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.362329 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353327    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353330    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353332    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353335    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353338    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353341    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353344    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353346    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353349    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353353    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353358    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353360    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353363    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353366    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353369    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353372    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353375    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353378    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353380    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353388    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.363051 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353391    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353394    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353396    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353399    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353402    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353406    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353410    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353413    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353416    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353419    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353422    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353425    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353428    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.353452    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353490    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:07.363552 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353495    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353498    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353501    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353504    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353507    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353509    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353512    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353516    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353520    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353523    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353526    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353529    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353531    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353534    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353537    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353540    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353542    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353545    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353547    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353550    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:07.363949 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353553    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353556    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353558    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353563    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353566    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353568    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353571    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353574    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353576    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353580    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353583    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353586    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353588    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353591    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353594    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353597    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353599    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353602    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353605    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:07.364461 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353608    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353610    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353613    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353615    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353618    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353620    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353623    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353626    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353629    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353631    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353634    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353636    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353642    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353645    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353647    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353650    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353652    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353655    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353658    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353660    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:07.364957 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353663    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353665    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353668    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353672    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353674    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353677    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353680    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353682    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353685    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353687    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353690    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353693    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353695    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353698    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353701    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353703    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353706    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353708    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353711    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353714    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:07.365547 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353716    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353719    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353723    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353725    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353729    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353733    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353736    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353739    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353741    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353744    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353747    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353749    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353752    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353755    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353757    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353760    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353763    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353766    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353768    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353771    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:07.366206 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353774    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353776    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353793    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353797    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353799    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:07.353803    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.353859    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.356358    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.357421    2579 server.go:1014] "Starting client certificate rotation"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.357522    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.357577    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.359746    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:51:07.366723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.361910    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:51:07.371609 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.371596    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:51:07.375667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.375654    2579 log.go:25] "Validated CRI v1 image API"
Nov 02 19:51:07.376473 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.376442    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:51:07.379357 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.379336    2579 fs.go:135] Filesystem UUIDs: map[43ba860f-ca0f-42dd-a575-f4a2be5f8455:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a3b183f-ef26-486d-b3e1-16df02ff4c0c:/dev/nvme0n1p3]
Nov 02 19:51:07.379434 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.379356    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 19:51:07.384722 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.384593    2579 manager.go:217] Machine: {Timestamp:2026-11-02 19:51:07.383114753 +0000 UTC m=+0.321291166 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087247 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9c89391440ee743e96a699027915 SystemUUID:ec2f9c89-3914-40ee-743e-96a699027915 BootID:dbf57f0f-a209-470a-89a6-49c1b920b0f7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c7:f3:4f:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c7:f3:4f:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:7c:9d:33:30:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:51:07.385379 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.385368    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:51:07.385466 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.385453    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:51:07.389292 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389269    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:51:07.389458 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389309    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:51:07.389501 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389461    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:51:07.389501 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389469    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:51:07.389550 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389503    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:51:07.389550 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.389521    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:51:07.390790 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.390770    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:51:07.395389 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.395377    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:51:07.395524 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.395514    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:51:07.397180 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397166    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:51:07.397246 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397197    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:51:07.397246 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397209    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:51:07.397246 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397218    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:51:07.397246 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397225    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:51:07.397668 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.397657    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:51:07.401147 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.400931    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:51:07.408366 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.408351    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:51:07.409270 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409255    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:51:07.409325 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409293    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:51:07.409485 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409474    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:51:07.409527 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409493    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:51:07.409527 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409502    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:51:07.409527 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409510    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:51:07.409527 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409518    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:51:07.409527 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.409526    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:51:07.410110 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410100    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:51:07.410141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410113    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:51:07.410141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410123    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:51:07.410141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410132    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:51:07.410211 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410144    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:51:07.410211 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.410156    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:51:07.411845 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.411834    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:51:07.411883 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.411847    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:51:07.416532 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.416517    2579 server.go:1263] "Started kubelet"
Nov 02 19:51:07.416622 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.416588    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:51:07.416702 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.416654    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:51:07.416742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.416730    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:51:07.418121 ip-10-0-133-26 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:51:07.418379 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.418344    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:51:07.419973 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.419959    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:51:07.424256 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.424216    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 19:51:07.427666 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.427645    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:51:07.429053 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429034    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:51:07.429151 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.429125    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:07.429861 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429841    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:51:07.429861 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429862    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:51:07.429980 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429882    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:51:07.430029 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429968    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:51:07.430029 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.429993    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:51:07.430457 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430438    2579 factory.go:55] Registering systemd factory
Nov 02 19:51:07.430561 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430461    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:51:07.430720 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430705    2579 factory.go:153] Registering CRI-O factory
Nov 02 19:51:07.430794 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430725    2579 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:51:07.430855 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430824    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:51:07.430908 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430859    2579 factory.go:103] Registering Raw factory
Nov 02 19:51:07.430908 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.430874    2579 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:51:07.432006 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.431991    2579 manager.go:319] Starting recovery of all containers
Nov 02 19:51:07.442461 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.442435    2579 manager.go:324] Recovery completed
Nov 02 19:51:07.448286 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.448268    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.450188 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450162    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.450257 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450203    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.450257 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450217    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.450868 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450851    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:51:07.450868 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450866    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:51:07.450967 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450882    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.450967 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.450884    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:51:07.451618 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.451605    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.451671 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.451633    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.451671 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.451643    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.453118 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.453105    2579 policy_none.go:50] "Start"
Nov 02 19:51:07.453192 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.453122    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:51:07.453192 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.453135    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:51:07.455080 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.455067    2579 policy_none.go:44] "Start"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.500410    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.504951    2579 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.504979    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.504991    2579 server.go:87] "Starting device plugin registration server"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.505240    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.505252    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.505337    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.505431    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.505440    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.505880    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.505929    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.509771    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.511434    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.511453    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.511485    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:51:07.511749 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.511525    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 19:51:07.547299 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.547217    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-26.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 19:51:07.547299 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.547251    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-26.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 19:51:07.548388 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.547209    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.us-west-2.compute.internal.18e45252db0f46e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:07.416307425 +0000 UTC m=+0.354483840,LastTimestamp:2026-11-02 19:51:07.416307425 +0000 UTC m=+0.354483840,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.us-west-2.compute.internal,}"
Nov 02 19:51:07.605427 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.605360    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.606327 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.606304    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.606411 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.606351    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.606411 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.606363    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.606411 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.606390    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.612586 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.612570    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"]
Nov 02 19:51:07.612631 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.612625    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.613421 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.613407    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.613468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.613432    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.613468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.613442    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.615451 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.615428    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-26.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 19:51:07.615527 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.615431    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 19:51:07.615740 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.615723    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 19:51:07.615839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.615827    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.615885 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.615770    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.us-west-2.compute.internal.18e45252dd143f6f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-26.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:07.450187631 +0000 UTC m=+0.388364044,LastTimestamp:2026-11-02 19:51:07.450187631 +0000 UTC m=+0.388364044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.us-west-2.compute.internal,}"
Nov 02 19:51:07.616014 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.615993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.616047 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616036    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.616596 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616579    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.616667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616597    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.616667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616606    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.616667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616617    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.616667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616620    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.616667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.616646    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.618685 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.618671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.618729 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.618709    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.619461 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.619446    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.619515 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.619473    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.619515 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.619483    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.640259 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.640241    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.642222 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.642207    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.647795 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.647766    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:51:07.647880 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.647824    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j4hrr"
Nov 02 19:51:07.693138 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.693000    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.us-west-2.compute.internal.18e45252dd149a1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-26.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:07.450210842 +0000 UTC m=+0.388387258,LastTimestamp:2026-11-02 19:51:07.450210842 +0000 UTC m=+0.388387258,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.us-west-2.compute.internal,}"
Nov 02 19:51:07.693311 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.693169    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 19:51:07.730860 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.730826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.731005 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.730868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.731005 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.730902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.762882 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.762854    2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-26.us-west-2.compute.internal\" not found" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831749 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831942 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831942 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831942 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.831942 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.831874    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.843098 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:07.843075    2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-26.us-west-2.compute.internal\" not found" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.945863 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.945804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.947965 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.947947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:07.963564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.963440    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:07.964657 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.964640    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:07.964753 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.964676    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:07.964753 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.964692    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:07.964753 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:07.964729    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:08.023689 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.023656    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:08.023689 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.023689    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-26.us-west-2.compute.internal\": node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.145302 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.145260    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.245579 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.245481    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.346663 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.346625    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.357586 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.357553    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 19:51:08.357869 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.357754    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a83af7d525ec44176aabcd0e4302dda2-d27cf1d047af92e6.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-26.us-west-2.compute.internal.18e45252dd143f6f\": read tcp 10.0.133.26:58490->3.140.89.17:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-26.us-west-2.compute.internal.18e45252dd143f6f  default   4801 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-26.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:07 +0000 UTC,LastTimestamp:2026-11-02 19:51:07.61342021 +0000 UTC m=+0.551596623,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.us-west-2.compute.internal,}"
Nov 02 19:51:08.447349 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.447317    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.548526 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.548440    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.628714 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.628669    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:08.630811 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.630777    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:08.630948 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.630824    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:08.630948 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.630840    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:08.648525 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.648495    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.653269 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.653228    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:08 +0000 UTC" deadline="2028-04-27 08:51:42.905742664 +0000 UTC"
Nov 02 19:51:08.653328 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.653269    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12997h0m34.252477139s"
Nov 02 19:51:08.701323 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.699705    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:08.701323 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.699719    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:08.701323 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.699745    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:08.749403 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.749365    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.849982 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:08.849882    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:08.863141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.863112    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:08.931692 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.931644    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:08.994053 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.994026    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:08.995246 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:08.995228    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:09.054200 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.054141    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:09.169868 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.169832    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba4ed11659b02193d9121e647db325.slice/crio-a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224 WatchSource:0}: Error finding container a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224: Status 404 returned error can't find the container with id a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224
Nov 02 19:51:09.170274 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.170250    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee94c0e3fb310e2d7d2c04225780f5ac.slice/crio-b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597 WatchSource:0}: Error finding container b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597: Status 404 returned error can't find the container with id b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597
Nov 02 19:51:09.178186 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.178169    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:51:09.400716 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.400674    2579 apiserver.go:52] "Watching apiserver"
Nov 02 19:51:09.430122 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.430031    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 19:51:09.506204 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.506173    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:51:09.506507 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.506488    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4drx6","openshift-image-registry/node-ca-s99qh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal","kube-system/konnectivity-agent-szw22","kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8","openshift-multus/multus-bv7hx","openshift-multus/network-metrics-daemon-rp4zx","openshift-network-diagnostics/network-check-target-f2l4l","openshift-network-operator/iptables-alerter-zp64p","openshift-ovn-kubernetes/ovnkube-node-hsmjq","kube-system/global-pull-secret-syncer-g5cr4","openshift-cluster-node-tuning-operator/tuned-b4m58","openshift-dns/node-resolver-4p96v"]
Nov 02 19:51:09.510525 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.510503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.512948 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.512917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.515280 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.515262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.518234 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.517657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.520143 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.520126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.520244 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.520183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:09.522521 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.522504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.524952 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.524933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:09.525044 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.524994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:09.527243 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.527224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.529874 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.529839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.532192 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.532174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.534743 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.534723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.537252 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.537233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.537353 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.537331    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:09.541408 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.541365    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"ee94c0e3fb310e2d7d2c04225780f5ac","Type":"ContainerStarted","Data":"b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597"}
Nov 02 19:51:09.541490 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.541420    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerStarted","Data":"a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224"}
Nov 02 19:51:09.542162 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.542265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.542265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542265 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.542386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.542386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.542386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.542386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:09.542564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.542564 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.542742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.542742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.542742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.542742 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.542986 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.542982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kw6\" (UniqueName: \"kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjb54\" (UniqueName: \"kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkgn\" (UniqueName: \"kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287d6\" (UniqueName: \"kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2987g\" (UniqueName: \"kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.543756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.543980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kzc\" (UniqueName: \"kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptshk\" (UniqueName: \"kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7whk\" (UniqueName: \"kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.544235 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.544765 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.544765 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.544212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.563588 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.563561    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.563694 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.563565    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.563694 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.563566    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:51:09.563694 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.563628    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:51:09.563814 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.563709    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\""
Nov 02 19:51:09.565059 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565036    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.565177 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565159    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\""
Nov 02 19:51:09.565231 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565192    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:51:09.565231 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565222    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:51:09.565335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565314    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:51:09.565391 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565363    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.565654 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.565641    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:51:09.567048 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.567035    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.567237 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.567225    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\""
Nov 02 19:51:09.567277 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.567228    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.569487 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.569474    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:51:09.569642 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.569626    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:51:09.569714 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.569702    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\""
Nov 02 19:51:09.574464 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.574449    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.610457 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.610425    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 19:51:09.610599 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.610471    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d8jqf"
Nov 02 19:51:09.630758 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.630720    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:51:09.644876 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.644954 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.644954 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.644954 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644949    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645088 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645088 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.644991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645088 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.645088 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645184    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645216    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645274 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645263    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kw6\" (UniqueName: \"kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.645443    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5h77\" (UniqueName: \"kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645510    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjb54\" (UniqueName: \"kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.645637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.645591    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:10.145564097 +0000 UTC m=+3.083740525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkgn\" (UniqueName: \"kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287d6\" (UniqueName: \"kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645890    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645923    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.646319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645913    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.645984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646042    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646105    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646141    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2987g\" (UniqueName: \"kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646248    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646302    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646387    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646497    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646581    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646591    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kzc\" (UniqueName: \"kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.647832 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646631    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptshk\" (UniqueName: \"kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7whk\" (UniqueName: \"kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646833    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646962    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.646995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647052    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.648817 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647327    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647444    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647473    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.649771 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647516    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647504    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647685    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647768    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647800    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.650432 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.647966    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.650270    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.650976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.650312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.650976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.650359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.650976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.650531    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.672857 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.672832    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.672975 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.672962    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:51:09.673061 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673046    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\""
Nov 02 19:51:09.673146 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673112    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:51:09.673146 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673127    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.673146 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673127    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673153    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673167    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673221    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673229    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673260    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673290    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673321    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:09.673452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673356    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.673729 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.673577    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\""
Nov 02 19:51:09.674030 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.674011    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:09.674078 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.674017    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:51:09.677703 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.677686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.678084 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.678068    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.702788 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.702726    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:09.702788 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.702754    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:09.703005 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.702796    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:09.703005 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.702878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:10.202841985 +0000 UTC m=+3.141018407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:09.704341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704313    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287d6\" (UniqueName: \"kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:09.704453 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2987g\" (UniqueName: \"kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.704453 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704393    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7whk\" (UniqueName: \"kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.704453 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704366    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptshk\" (UniqueName: \"kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.704624 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704476    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkgn\" (UniqueName: \"kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.704774 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.704753    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjb54\" (UniqueName: \"kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.705126 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.705108    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kw6\" (UniqueName: \"kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.705455 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.705439    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kzc\" (UniqueName: \"kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.748136 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748283 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h77\" (UniqueName: \"kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748283 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.748283 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748283 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748283 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.748307    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748354    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748365    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:09.748393    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:10.24837059 +0000 UTC m=+3.186547011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.748575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748850 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748583    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748850 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:09.748850 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748633    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748850 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748738    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748919    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.748976 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.748952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.804991 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.804956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h77\" (UniqueName: \"kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.824022 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.823990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:09.835255 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.835230    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d28c42_9ad6_4a4f_a070_19a056e3bd19.slice/crio-40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79 WatchSource:0}: Error finding container 40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79: Status 404 returned error can't find the container with id 40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79
Nov 02 19:51:09.841838 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.841819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:09.849949 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.849930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:09.850114 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.850098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:09.850230 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.850205    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499f1915_5219_4094_9256_e01f23a3545b.slice/crio-7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295 WatchSource:0}: Error finding container 7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295: Status 404 returned error can't find the container with id 7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295
Nov 02 19:51:09.857238 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.857221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:09.859434 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.859416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:09.860697 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.860677    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeed0e8_d128_4f10_b30c_f8930fc0837b.slice/crio-2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be WatchSource:0}: Error finding container 2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be: Status 404 returned error can't find the container with id 2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be
Nov 02 19:51:09.862938 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.862881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:09.863587 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.863561    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22ccf89_c8b1_46c3_a6ae_feaecb3a180f.slice/crio-46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135 WatchSource:0}: Error finding container 46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135: Status 404 returned error can't find the container with id 46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135
Nov 02 19:51:09.868634 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.868614    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490663ba_67e7_464e_ab56_9e5d2fdb1ca8.slice/crio-8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf WatchSource:0}: Error finding container 8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf: Status 404 returned error can't find the container with id 8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf
Nov 02 19:51:09.870836 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.870815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:09.871732 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.871713    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4383d7_9852_4248_830b_1163ddc9a177.slice/crio-4e22faf42fc0a699f3c467a794c30dea65a90fca87a5fa1cf2369952f7cf49de WatchSource:0}: Error finding container 4e22faf42fc0a699f3c467a794c30dea65a90fca87a5fa1cf2369952f7cf49de: Status 404 returned error can't find the container with id 4e22faf42fc0a699f3c467a794c30dea65a90fca87a5fa1cf2369952f7cf49de
Nov 02 19:51:09.874632 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.874583    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf393cf71_bee3_472a_aa0f_e2be7e8c722a.slice/crio-7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202 WatchSource:0}: Error finding container 7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202: Status 404 returned error can't find the container with id 7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202
Nov 02 19:51:09.881037 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.881017    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b9b178_30a3_4be9_ae42_663579a9bbab.slice/crio-c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6 WatchSource:0}: Error finding container c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6: Status 404 returned error can't find the container with id c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6
Nov 02 19:51:09.889202 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:09.889178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:09.898365 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:09.898343    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d101f8_edc6_402a_a34c_922529a5b226.slice/crio-f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1 WatchSource:0}: Error finding container f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1: Status 404 returned error can't find the container with id f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1
Nov 02 19:51:10.150818 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.150695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:10.151042 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.150941    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:10.151042 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.151010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:11.150991857 +0000 UTC m=+4.089168265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:10.251346 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.251304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:10.251555 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.251389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:10.251620 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251606    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:10.251673 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251627    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:10.251673 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251641    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:10.251776 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251703    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:11.251681883 +0000 UTC m=+4.189858288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:10.251897 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251876    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:10.251964 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:10.251939    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:11.251925155 +0000 UTC m=+4.190101566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:10.522189 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.522108    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerStarted","Data":"f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1"}
Nov 02 19:51:10.524280 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.524242    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zp64p" event={"ID":"26b9b178-30a3-4be9-ae42-663579a9bbab","Type":"ContainerStarted","Data":"c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6"}
Nov 02 19:51:10.525777 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.525749    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s99qh" event={"ID":"f393cf71-bee3-472a-aa0f-e2be7e8c722a","Type":"ContainerStarted","Data":"7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202"}
Nov 02 19:51:10.527149 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.527121    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4m58" event={"ID":"9c4383d7-9852-4248-830b-1163ddc9a177","Type":"ContainerStarted","Data":"4e22faf42fc0a699f3c467a794c30dea65a90fca87a5fa1cf2369952f7cf49de"}
Nov 02 19:51:10.528470 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.528445    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szw22" event={"ID":"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f","Type":"ContainerStarted","Data":"46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135"}
Nov 02 19:51:10.529769 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.529745    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be"}
Nov 02 19:51:10.531719 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.531696    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8" event={"ID":"490663ba-67e7-464e-ab56-9e5d2fdb1ca8","Type":"ContainerStarted","Data":"8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf"}
Nov 02 19:51:10.533331 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.533308    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4p96v" event={"ID":"499f1915-5219-4094-9256-e01f23a3545b","Type":"ContainerStarted","Data":"7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295"}
Nov 02 19:51:10.534540 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.534515    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bv7hx" event={"ID":"02d28c42-9ad6-4a4f-a070-19a056e3bd19","Type":"ContainerStarted","Data":"40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79"}
Nov 02 19:51:10.614078 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.614034    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:09 +0000 UTC" deadline="2028-08-03 01:09:59.67878998 +0000 UTC"
Nov 02 19:51:10.614078 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:10.614069    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15341h18m49.064726534s"
Nov 02 19:51:11.158458 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.158420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:11.158645 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.158558    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:11.158645 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.158608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:13.158594956 +0000 UTC m=+6.096771363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:11.259180 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.259134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:11.259377 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.259225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:11.259440 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259399    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:11.259440 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259418    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:11.259440 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259431    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:11.259602 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:13.25947364 +0000 UTC m=+6.197650046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:11.259602 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259573    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:11.259712 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.259610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:13.259598453 +0000 UTC m=+6.197774858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:11.513145 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.513057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:11.514615 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.513980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:11.514615 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.514169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:11.514615 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.514267    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:11.514615 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.514324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:11.514615 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:11.514411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:11.615093 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.615021    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:09 +0000 UTC" deadline="2028-08-10 03:01:27.387056262 +0000 UTC"
Nov 02 19:51:11.615093 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:11.615090    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15511h10m15.771971783s"
Nov 02 19:51:13.177041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.176690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:13.177577 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.176921    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:13.177577 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.177159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:17.177109383 +0000 UTC m=+10.115285810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.277508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.277554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277687    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277702    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277712    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277757    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:17.277743886 +0000 UTC m=+10.215920302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277812    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:13.277886 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.277855    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:17.277845397 +0000 UTC m=+10.216021797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.512045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.512155    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.512205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.512276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.512305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:13.514007 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:13.512343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:13.555551 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.554764    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"ee94c0e3fb310e2d7d2c04225780f5ac","Type":"ContainerStarted","Data":"7f1356af0cac5d0817c757036310db548c8f6a9c2ad9e0cafe52bbc8435b1091"}
Nov 02 19:51:13.668851 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:13.668042    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal" podStartSLOduration=4.668024858 podStartE2EDuration="4.668024858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:13.667552783 +0000 UTC m=+6.605729210" watchObservedRunningTime="2026-11-02 19:51:13.668024858 +0000 UTC m=+6.606201285"
Nov 02 19:51:14.573438 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:14.572773    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="68c31f946197bb6cf941081a83d7e8da5d7a2c85088f9a989ae94505b4a7b23b" exitCode=0
Nov 02 19:51:14.573438 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:14.572907    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"68c31f946197bb6cf941081a83d7e8da5d7a2c85088f9a989ae94505b4a7b23b"}
Nov 02 19:51:14.582324 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:14.581271    2579 generic.go:356] "Generic (PLEG): container finished" podID="9bba4ed11659b02193d9121e647db325" containerID="51d0aedee076c669c46a8bfb27f3340de15b3e9d0d403bb29261cd0bf2cf0a70" exitCode=0
Nov 02 19:51:14.582324 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:14.581878    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerDied","Data":"51d0aedee076c669c46a8bfb27f3340de15b3e9d0d403bb29261cd0bf2cf0a70"}
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.511972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:15.512121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.512553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:15.512647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.512711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:15.515041 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:15.512808    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:15.585609 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.585565    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szw22" event={"ID":"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f","Type":"ContainerStarted","Data":"0a940b6751a5a50440d30b6dd85fd0c0236adbee1f58b00eec21bca7bb9be355"}
Nov 02 19:51:15.598055 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.597558    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerStarted","Data":"fcb3944cee8d9af4b14052a0cb60080d40a140dceb9dd1fb02ef7df5bbca3d68"}
Nov 02 19:51:15.795259 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.795154    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" podStartSLOduration=6.795133211 podStartE2EDuration="6.795133211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:15.794831253 +0000 UTC m=+8.733007678" watchObservedRunningTime="2026-11-02 19:51:15.795133211 +0000 UTC m=+8.733309637"
Nov 02 19:51:15.795430 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:15.795248    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-szw22" podStartSLOduration=2.849068451 podStartE2EDuration="7.795241047s" totalImagesPullingTime="4.946172596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:15.691143074 +0000 UTC m=+8.629319497" watchObservedRunningTime="2026-11-02 19:51:15.795241047 +0000 UTC m=+8.733417469"
Nov 02 19:51:16.590109 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.589999    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:16.590757 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.590740    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:16.602684 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.602658    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s99qh" event={"ID":"f393cf71-bee3-472a-aa0f-e2be7e8c722a","Type":"ContainerStarted","Data":"490350208a60ce1687184b6c1835b47c84900f0f811564bf3489f979222597d8"}
Nov 02 19:51:16.798023 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.797968    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s99qh" podStartSLOduration=2.541517866 podStartE2EDuration="8.797946904s" totalImagesPullingTime="6.256429038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:16.797289454 +0000 UTC m=+9.735465880" watchObservedRunningTime="2026-11-02 19:51:16.797946904 +0000 UTC m=+9.736123328"
Nov 02 19:51:16.960062 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.959975    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:16.960826 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:16.960607    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:17.215580 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.214986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:17.215580 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.215108    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:17.215580 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.215156    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:25.215143042 +0000 UTC m=+18.153319443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:17.315553 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.315513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:17.315854 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.315608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:17.315854 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315774    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:17.315854 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315805    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:17.315854 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315821    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:17.316112 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:25.315859783 +0000 UTC m=+18.254036189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:17.316112 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315926    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:17.316112 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.315982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:25.315967793 +0000 UTC m=+18.254144206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.512488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.512571    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.512955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.513068    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.513123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:17.514297 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:17.513169    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:17.609063 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:17.609023    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8" event={"ID":"490663ba-67e7-464e-ab56-9e5d2fdb1ca8","Type":"ContainerStarted","Data":"9ca5842b9e10c93208f92921959df3f6e06de7b28cd7ebc2612f0f5c99c7c096"}
Nov 02 19:51:18.614642 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:18.614610    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bv7hx" event={"ID":"02d28c42-9ad6-4a4f-a070-19a056e3bd19","Type":"ContainerStarted","Data":"b22c876fef0b56b85f05dc0e4d9762ce07b4d2df7630a22145b833c5b1da35d1"}
Nov 02 19:51:18.721907 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:18.721661    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bv7hx" podStartSLOduration=2.245445649 podStartE2EDuration="10.721644675s" totalImagesPullingTime="8.476199026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:18.721177316 +0000 UTC m=+11.659353738" watchObservedRunningTime="2026-11-02 19:51:18.721644675 +0000 UTC m=+11.659821096"
Nov 02 19:51:18.840173 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:18.840150    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:51:19.511119 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.510991    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:51:18.840169359Z","UUID":"7f8bca84-5e85-4c2b-ab57-af9fc375dc94","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:51:19.513080 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.513049    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:51:19.513080 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.513085    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.515421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.515451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.515454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:19.515556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:19.515625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:19.515928 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:19.515754    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:19.619384 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.619351    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8" event={"ID":"490663ba-67e7-464e-ab56-9e5d2fdb1ca8","Type":"ContainerStarted","Data":"3ce83aab8aab1c71d43155775c1176bada65247b6bc1364731bf24a56e51413c"}
Nov 02 19:51:19.621700 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.621289    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4p96v" event={"ID":"499f1915-5219-4094-9256-e01f23a3545b","Type":"ContainerStarted","Data":"158e556c7f52cbf6e56d2e6ccf00c327bd53b916ef4c1a88872dcbcb57814493"}
Nov 02 19:51:19.623587 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.623565    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="0130e176c1d0ae9c15393aa15cf5b082fa1fed4310eb5cc99811d155e44aefcd" exitCode=0
Nov 02 19:51:19.623689 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.623659    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"0130e176c1d0ae9c15393aa15cf5b082fa1fed4310eb5cc99811d155e44aefcd"}
Nov 02 19:51:19.727585 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:19.727537    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4p96v" podStartSLOduration=2.919049545 podStartE2EDuration="11.727517464s" totalImagesPullingTime="8.808467919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:19.726917712 +0000 UTC m=+12.665094134" watchObservedRunningTime="2026-11-02 19:51:19.727517464 +0000 UTC m=+12.665693886"
Nov 02 19:51:20.628701 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:20.628654    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8" event={"ID":"490663ba-67e7-464e-ab56-9e5d2fdb1ca8","Type":"ContainerStarted","Data":"5d037155b59527f43f5cc3bd137619024e36dcb4d1f2de45a5ed7cbcb4a26e16"}
Nov 02 19:51:20.630321 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:20.630293    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zp64p" event={"ID":"26b9b178-30a3-4be9-ae42-663579a9bbab","Type":"ContainerStarted","Data":"fb5df0eaba11ecd58c2f351489ada2c8b5b03d38fceae9b14edf7508cc84ff0c"}
Nov 02 19:51:20.735328 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:20.735275    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8" podStartSLOduration=2.720184769 podStartE2EDuration="12.73526097s" totalImagesPullingTime="10.015076201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:20.734944955 +0000 UTC m=+13.673121376" watchObservedRunningTime="2026-11-02 19:51:20.73526097 +0000 UTC m=+13.673437392"
Nov 02 19:51:20.839750 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:20.839697    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zp64p" podStartSLOduration=4.513763563 podStartE2EDuration="12.839682989s" totalImagesPullingTime="8.325919426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:20.839055562 +0000 UTC m=+13.777231985" watchObservedRunningTime="2026-11-02 19:51:20.839682989 +0000 UTC m=+13.777859412"
Nov 02 19:51:21.512269 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.512190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:21.512269 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.512231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:21.512485 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:21.512317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:21.512485 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:21.512362    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:21.512485 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.512415    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:21.512485 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:21.512479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:21.636705 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.636675    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="2421d86f4c0a3ed7cef161a77eed1653eecc494b07302d5590cf20a8065ea558" exitCode=0
Nov 02 19:51:21.637223 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.636750    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"2421d86f4c0a3ed7cef161a77eed1653eecc494b07302d5590cf20a8065ea558"}
Nov 02 19:51:21.639420 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:21.639397    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4m58" event={"ID":"9c4383d7-9852-4248-830b-1163ddc9a177","Type":"ContainerStarted","Data":"ef35d2f7d778f5658eb656ab2d49cf970d1ee65bd0f3ee576149e5788d981dff"}
Nov 02 19:51:23.512563 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.512346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:23.513230 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.512363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:23.513230 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:23.512678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:23.513230 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:23.512731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:23.513230 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.512384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:23.513230 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:23.512829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:23.646370 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.646330    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="2b441afcd9cc47b1ebfcce515193243ee210143efc9ff90b032956c60d5ccd04" exitCode=0
Nov 02 19:51:23.646544 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.646374    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"2b441afcd9cc47b1ebfcce515193243ee210143efc9ff90b032956c60d5ccd04"}
Nov 02 19:51:23.848839 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:23.848716    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b4m58" podStartSLOduration=4.83771125 podStartE2EDuration="15.848703497s" totalImagesPullingTime="11.010992247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:21.854056556 +0000 UTC m=+14.792232990" watchObservedRunningTime="2026-11-02 19:51:23.848703497 +0000 UTC m=+16.786879918"
Nov 02 19:51:25.275973 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.275939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:25.276489 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.276131    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:25.276489 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.276213    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.27619185 +0000 UTC m=+34.214368264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 19:51:25.377109 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.377081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:25.377234 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.377222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:25.377301 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377237    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 19:51:25.377301 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377268    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 19:51:25.377301 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377281    2579 projected.go:196] Error preparing data for projected volume kube-api-access-frggk for pod openshift-network-diagnostics/network-check-target-f2l4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:25.377404 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377323    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:25.377404 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377340    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk podName:e1b91d97-3140-4872-9e51-68cc0ad1de35 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.377322196 +0000 UTC m=+34.315498610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frggk" (UniqueName: "kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk") pod "network-check-target-f2l4l" (UID: "e1b91d97-3140-4872-9e51-68cc0ad1de35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 19:51:25.377404 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.377373    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret podName:b64f0c97-c8f6-4e42-ac7d-32b5d6694968 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.377359511 +0000 UTC m=+34.315535914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret") pod "global-pull-secret-syncer-g5cr4" (UID: "b64f0c97-c8f6-4e42-ac7d-32b5d6694968") : object "kube-system"/"original-pull-secret" not registered
Nov 02 19:51:25.512312 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.512291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:25.512435 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.512345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:25.512435 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.512358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:25.512552 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.512445    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:25.512552 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.512523    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:25.512641 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:25.512617    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:25.655830 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.655583    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"2212ff03fbc1c7cc71589469f6e9eb3ed72df64a53cc55d027d14d4bee8fa44c"}
Nov 02 19:51:25.655947 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.655896    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"a58883dc52aa220dbea23cc932c414064d3b0ba465c64f49888b180409f11c9f"}
Nov 02 19:51:25.656010 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.655946    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"c4dbc8da810c030f3aa83c913d1d86997d109c40ef1b363edbd9542a3b41e8b9"}
Nov 02 19:51:25.656010 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.655992    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"a94c1cb3b51fb32b74c91188391e0c23899b61f30550f38a6eaaaf225a774759"}
Nov 02 19:51:25.656095 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:25.656039    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"7e971d38d4ef9e87d390a3f7567dc7311a785125f37dc4ede17f04e477bee691"}
Nov 02 19:51:26.660683 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:26.660647    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"268dc317cba7d26ae4ef7f79d59713db38bfb8b7db84217ec44722046c4a88ba"}
Nov 02 19:51:26.663193 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:26.663170    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="4f2cd70c93e56697f4916257c56581b0bf129c61b5d5139585fef73f00d70c40" exitCode=0
Nov 02 19:51:26.663308 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:26.663205    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"4f2cd70c93e56697f4916257c56581b0bf129c61b5d5139585fef73f00d70c40"}
Nov 02 19:51:27.512760 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:27.512726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:27.512954 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:27.512835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:27.512954 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:27.512863    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:27.512954 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:27.512881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:27.513104 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:27.512957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:27.513104 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:27.513047    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:27.667706 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:27.667678    2579 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="d0c466670645cd6a70ca5f5bd7abd83a9e90c581859c4c84d60c4420befbd92d" exitCode=0
Nov 02 19:51:27.668160 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:27.667716    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerDied","Data":"d0c466670645cd6a70ca5f5bd7abd83a9e90c581859c4c84d60c4420befbd92d"}
Nov 02 19:51:28.673042 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:28.673006    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"9f7e4c93afd049f929bc1bcef921e79fbac00d1d06d3c205e13dba9e6d3790eb"}
Nov 02 19:51:28.676193 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:28.676171    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drx6" event={"ID":"c9d101f8-edc6-402a-a34c-922529a5b226","Type":"ContainerStarted","Data":"e59891e8fa0570a069b705285cf15e10b7149b485d22a0f14ff60084c7cbe724"}
Nov 02 19:51:28.880723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:28.880653    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4drx6" podStartSLOduration=7.612146708 podStartE2EDuration="20.880639659s" totalImagesPullingTime="13.268492951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:28.88029279 +0000 UTC m=+21.818469213" watchObservedRunningTime="2026-11-02 19:51:28.880639659 +0000 UTC m=+21.818816081"
Nov 02 19:51:29.514420 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:29.514392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:29.514588 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:29.514474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:29.514588 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:29.514580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:29.514716 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:29.514596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:29.514716 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:29.514660    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:29.514716 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:29.514702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:30.684440 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:30.684407    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" event={"ID":"fbeed0e8-d128-4f10-b30c-f8930fc0837b","Type":"ContainerStarted","Data":"aeafe3e864c10e6c65339b031a525cb9b0bbfaf8b73ecfb9eaaa8ad77ac24e03"}
Nov 02 19:51:30.685120 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:30.684646    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:30.699923 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:30.699901    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:30.935622 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:30.935533    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq" podStartSLOduration=7.4858701100000005 podStartE2EDuration="22.935518755s" totalImagesPullingTime="15.449648645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:30.935191611 +0000 UTC m=+23.873368032" watchObservedRunningTime="2026-11-02 19:51:30.935518755 +0000 UTC m=+23.873695178"
Nov 02 19:51:31.511728 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.511697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:31.511728 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.511717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:31.512022 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:31.511882    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:31.512022 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.511924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:31.512022 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:31.512003    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:31.512113 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:31.512061    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:31.597088 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.597058    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:31.686623 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.686596    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:31.701027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:31.701006    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:32.382764 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.382735    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rp4zx"]
Nov 02 19:51:32.382925 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.382853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:32.382969 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:32.382951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:32.385160 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.385136    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5cr4"]
Nov 02 19:51:32.385300 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.385233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:32.385370 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:32.385343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:32.385756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.385735    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f2l4l"]
Nov 02 19:51:32.385847 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:32.385835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:32.385957 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:32.385932    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:33.512407 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:33.512376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:33.512834 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:33.512489    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:34.511852 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:34.511820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:34.511852 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:34.511835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:34.512063 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:34.511929    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:34.512063 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:34.512027    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:35.512463 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:35.512434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:35.512940 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:35.512540    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:36.512248 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:36.512218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:36.512418 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:36.512313    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:36.512418 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:36.512370    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:36.512500 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:36.512459    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:37.512467 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:37.512436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:37.512658 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:37.512537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rp4zx" podUID="cffe96fe-db7d-4362-bb90-25363f34503a"
Nov 02 19:51:38.512491 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:38.512455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:38.512691 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:38.512455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:38.512691 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:38.512588    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5cr4" podUID="b64f0c97-c8f6-4e42-ac7d-32b5d6694968"
Nov 02 19:51:38.512691 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:38.512638    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f2l4l" podUID="e1b91d97-3140-4872-9e51-68cc0ad1de35"
Nov 02 19:51:39.377559 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.377522    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeReady"
Nov 02 19:51:39.377745 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.377675    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 19:51:39.482979 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.482942    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wbhwp"]
Nov 02 19:51:39.485723 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.485707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:39.511813 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.511772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:39.522931 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.522909    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sk775"]
Nov 02 19:51:39.525324 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.525308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:39.529480 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.529448    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kst75"]
Nov 02 19:51:39.532047 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.532030    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:51:39.532155 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.532136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:39.534539 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.534524    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"]
Nov 02 19:51:39.534641 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.534627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.536997 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.536981    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-chn6b"]
Nov 02 19:51:39.537122 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.537104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.539553 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.539529    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddb8948dc-fpq67"]
Nov 02 19:51:39.539729 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.539595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.543106 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.543089    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"]
Nov 02 19:51:39.543230 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.543215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.545833 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.545738    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"]
Nov 02 19:51:39.545897 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.545871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:39.548520 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.548508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.571403 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.571380    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"]
Nov 02 19:51:39.574255 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.574226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.575489 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.575387    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\""
Nov 02 19:51:39.575489 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.575418    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.575489 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.575459    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:51:39.575741 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.575699    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.575741 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.575736    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"]
Nov 02 19:51:39.580243 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.579970    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s5vct"]
Nov 02 19:51:39.580823 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.580775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.582939 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.582918    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:51:39.583065 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.583049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.583508 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.583479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqscx\" (UniqueName: \"kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx\") pod \"kas-connection-checker-685895c7c9-wbhwp\" (UID: \"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc\") " pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:39.585044 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.585021    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 19:51:39.585162 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.585115    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 19:51:39.586326 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.586302    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"]
Nov 02 19:51:39.586486 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.586398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.593324 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.593289    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"]
Nov 02 19:51:39.593464 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.593402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.597625 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.597595    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4z8z6"]
Nov 02 19:51:39.597812 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.597768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:39.600616 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.600594    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wm7kv"]
Nov 02 19:51:39.600736 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.600694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.604319 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604296    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wbhwp"]
Nov 02 19:51:39.604425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604338    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sk775"]
Nov 02 19:51:39.604425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604365    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kst75"]
Nov 02 19:51:39.604425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604387    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"]
Nov 02 19:51:39.604425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604408    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-chn6b"]
Nov 02 19:51:39.604425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.604682 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604429    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"]
Nov 02 19:51:39.604682 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604600    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"]
Nov 02 19:51:39.604682 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.604626    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"]
Nov 02 19:51:39.617644 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.617624    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"]
Nov 02 19:51:39.621837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.621817    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"]
Nov 02 19:51:39.622524 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.622507    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:51:39.623237 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.623218    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"]
Nov 02 19:51:39.634146 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.634126    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:51:39.637825 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.637802    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddb8948dc-fpq67"]
Nov 02 19:51:39.640467 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640446    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.640467 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640456    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.640634 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640464    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 19:51:39.640634 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640606    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.640634 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640612    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.640855 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.640838    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 19:51:39.641560 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.641538    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 19:51:39.641776 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.641751    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.643391 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643367    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.643482 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643432    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.643546 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643509    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.643629 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643615    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.643745 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643733    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.643924 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.643907    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 19:51:39.644034 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.644020    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.644263 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.644247    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 19:51:39.644969 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.644929    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.645315 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.645141    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.645315 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.645181    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.645501 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.645348    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.646736 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.646576    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.646837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.646724    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 19:51:39.684122 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpstz\" (UniqueName: \"kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz\") pod \"volume-data-source-validator-58c9f5f46d-6btrs\" (UID: \"d4169626-adde-4b36-85e9-11a8c5bb2626\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:39.684122 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwr48\" (UniqueName: \"kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c888\" (UniqueName: \"kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888\") pod \"network-check-source-6f78d59f99-xxjw5\" (UID: \"8fa9b69c-543e-49dd-b4f9-64c293ea4255\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.684386 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.684663 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.684663 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.684663 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwfv\" (UniqueName: \"kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.684663 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.684663 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.684916 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqscx\" (UniqueName: \"kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx\") pod \"kas-connection-checker-685895c7c9-wbhwp\" (UID: \"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc\") " pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:39.684916 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.684916 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.684916 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.684916 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.684962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.685144 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.685329 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnp7\" (UniqueName: \"kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7\") pod \"kas-connection-checker-685895c7c9-sk775\" (UID: \"16088935-ac22-423a-b525-b5051055e3dc\") " pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:39.685329 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.685329 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.685329 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.685329 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.685468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vs8\" (UniqueName: \"kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.685468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.685468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.685468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.685468 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27rl\" (UniqueName: \"kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.685645 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.685645 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.685645 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.685645 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.685837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4db\" (UniqueName: \"kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.685837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdh88\" (UniqueName: \"kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.685837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.685837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.685837 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2q9\" (UniqueName: \"kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75r4\" (UniqueName: \"kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.685956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.686018 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.686275 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pln4f\" (UniqueName: \"kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f\") pod \"kas-connection-checker-685895c7c9-kst75\" (UID: \"e9711f2a-a824-4a90-afa0-9c2bd6661f6a\") " pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:39.686275 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.686275 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.686275 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.686275 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.686479 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.686344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.687401 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687358    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 19:51:39.687604 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687575    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 19:51:39.687865 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687839    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 19:51:39.687865 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687858    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687901    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687904    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687903    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687926    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687929    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687940    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687962    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687950    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.687992    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:51:39.688002 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688002    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.688341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688009    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:51:39.688341 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688010    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k984j\""
Nov 02 19:51:39.688482 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688470    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5szdl\""
Nov 02 19:51:39.688521 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688479    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f686f\""
Nov 02 19:51:39.688927 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.688915    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4z8z6"]
Nov 02 19:51:39.689548 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.689532    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s5vct"]
Nov 02 19:51:39.690382 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690370    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gf92x\""
Nov 02 19:51:39.690424 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690415    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-45fn9\""
Nov 02 19:51:39.690463 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690434    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hphz2\""
Nov 02 19:51:39.690463 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690458    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nrwfl\""
Nov 02 19:51:39.690529 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690474    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xq6qx\""
Nov 02 19:51:39.690529 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690487    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fnfjn\""
Nov 02 19:51:39.690588 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.690541    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4dvwg\""
Nov 02 19:51:39.709760 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.709729    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\""
Nov 02 19:51:39.709919 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.709832    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\""
Nov 02 19:51:39.713577 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.713556    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 19:51:39.714559 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.714541    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wm7kv"]
Nov 02 19:51:39.729327 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.729305    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 19:51:39.729379 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.729344    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 19:51:39.729446 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.729431    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 19:51:39.738587 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.738557    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqscx\" (UniqueName: \"kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx\") pod \"kas-connection-checker-685895c7c9-wbhwp\" (UID: \"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc\") " pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:39.740991 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.740962    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.741100 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.741010    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:51:39.741100 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.741030    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:51:39.741100 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.741094    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:51:39.787032 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.786998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.787032 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.787287 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.787287 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.787287 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpstz\" (UniqueName: \"kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz\") pod \"volume-data-source-validator-58c9f5f46d-6btrs\" (UID: \"d4169626-adde-4b36-85e9-11a8c5bb2626\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:39.787444 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr48\" (UniqueName: \"kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.787444 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.787444 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.787365    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.287344793 +0000 UTC m=+33.225521207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:39.787444 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.787444 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c888\" (UniqueName: \"kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888\") pod \"network-check-source-6f78d59f99-xxjw5\" (UID: \"8fa9b69c-543e-49dd-b4f9-64c293ea4255\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwfv\" (UniqueName: \"kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.787681 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnp7\" (UniqueName: \"kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7\") pod \"kas-connection-checker-685895c7c9-sk775\" (UID: \"16088935-ac22-423a-b525-b5051055e3dc\") " pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.787970    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.787981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.788027    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.288008932 +0000 UTC m=+33.226185340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:39.788141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vs8\" (UniqueName: \"kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27rl\" (UniqueName: \"kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.788937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.788992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4db\" (UniqueName: \"kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh88\" (UniqueName: \"kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2q9\" (UniqueName: \"kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75r4\" (UniqueName: \"kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.789452 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pln4f\" (UniqueName: \"kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f\") pod \"kas-connection-checker-685895c7c9-kst75\" (UID: \"e9711f2a-a824-4a90-afa0-9c2bd6661f6a\") " pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.789956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.790048    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.790062    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:39.790166 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.790138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.290092868 +0000 UTC m=+33.228269272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:39.790454 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.790279    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.790578    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791113    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.791650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.791711    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:39.791909 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.791758    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.291742641 +0000 UTC m=+33.229919054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:39.792425 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.792110    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:39.792425 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.792149    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.292139056 +0000 UTC m=+33.230315456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:39.792425 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.792200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.792579 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.792481    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.792876    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.792979    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.792994    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.793039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.293022572 +0000 UTC m=+33.231198987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.793078    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.793372    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.793528    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.793904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794138    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.794157    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.794206    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.294190926 +0000 UTC m=+33.232367349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794218    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.794294    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:39.794331    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.294319198 +0000 UTC m=+33.232495633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:39.794474 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794477    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.795335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.795335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794752    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.795335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794774    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.795335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.794942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.795335 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.795054    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.796973 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.796953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:39.840075 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.840006    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr48\" (UniqueName: \"kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.840075 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.840065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpstz\" (UniqueName: \"kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz\") pod \"volume-data-source-validator-58c9f5f46d-6btrs\" (UID: \"d4169626-adde-4b36-85e9-11a8c5bb2626\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:39.843418 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.843310    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vs8\" (UniqueName: \"kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:39.843548 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.843524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c888\" (UniqueName: \"kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888\") pod \"network-check-source-6f78d59f99-xxjw5\" (UID: \"8fa9b69c-543e-49dd-b4f9-64c293ea4255\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:39.846347 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846299    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27rl\" (UniqueName: \"kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:39.846478 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4db\" (UniqueName: \"kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:39.846478 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnp7\" (UniqueName: \"kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7\") pod \"kas-connection-checker-685895c7c9-sk775\" (UID: \"16088935-ac22-423a-b525-b5051055e3dc\") " pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:39.846478 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846450    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwfv\" (UniqueName: \"kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:39.846838 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846820    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh88\" (UniqueName: \"kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.846987 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.846946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.847682 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.847615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.849061 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.848562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2q9\" (UniqueName: \"kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:39.849061 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.848587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.849061 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.849026    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:39.849904 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.849611    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75r4\" (UniqueName: \"kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.849904 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.849877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:39.850545 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.850514    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pln4f\" (UniqueName: \"kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f\") pod \"kas-connection-checker-685895c7c9-kst75\" (UID: \"e9711f2a-a824-4a90-afa0-9c2bd6661f6a\") " pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:39.864085 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.864049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:39.870575 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.870548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:39.886902 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.886880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:39.898014 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.897984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:39.915297 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.915232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:39.935299 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.934991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:39.946149 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:39.946125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:40.136924 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.136826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:40.144962 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.144936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:40.168473 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.168442    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wbhwp"]
Nov 02 19:51:40.171911 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.171886    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea63f5a8_8b6a_4cd4_9cca_f0b4e4473acc.slice/crio-26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411 WatchSource:0}: Error finding container 26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411: Status 404 returned error can't find the container with id 26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411
Nov 02 19:51:40.248133 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.248108    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"]
Nov 02 19:51:40.250745 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.250707    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-chn6b"]
Nov 02 19:51:40.259280 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.259252    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b6384d_6d2b_416b_900d_30cfa9f6ca93.slice/crio-bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5 WatchSource:0}: Error finding container bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5: Status 404 returned error can't find the container with id bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5
Nov 02 19:51:40.300494 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.299264    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:40.300494 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.299345    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.299324738 +0000 UTC m=+34.237501139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:40.300494 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.300402    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"]
Nov 02 19:51:40.301100 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.301035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:40.301491 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.301396    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:40.301491 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.301456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.301436888 +0000 UTC m=+34.239613303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:40.301679 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.301657    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"]
Nov 02 19:51:40.301805 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.301762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:40.301971 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.301939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:40.302138 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.302091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:40.302372 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.302306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:40.302372 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.302345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:40.302932 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.302829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:40.302932 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.302899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:40.303412 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.302602    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:40.303582 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.303538    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:40.303739 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.302663    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:40.303739 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.303688    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:40.303739 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.302710    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:40.304038 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.303964    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:40.304038 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.302758    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:40.304239 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.303302    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.303286993 +0000 UTC m=+34.241463396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:40.304239 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.304201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.304184515 +0000 UTC m=+34.242360920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:40.304239 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.304216    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.30420744 +0000 UTC m=+34.242383904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:40.304510 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.304449    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.304436706 +0000 UTC m=+34.242613112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:40.304510 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.304470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.30446116 +0000 UTC m=+34.242637564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:40.304510 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:40.304484    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.30447617 +0000 UTC m=+34.242652576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:40.308531 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.308507    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f9f30c_31b6_4d93_b608_19cfad05e344.slice/crio-b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4 WatchSource:0}: Error finding container b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4: Status 404 returned error can't find the container with id b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4
Nov 02 19:51:40.309056 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.309036    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4169626_adde_4b36_85e9_11a8c5bb2626.slice/crio-39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320 WatchSource:0}: Error finding container 39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320: Status 404 returned error can't find the container with id 39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320
Nov 02 19:51:40.331068 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.331036    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-s5vct"]
Nov 02 19:51:40.334727 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.332150    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"]
Nov 02 19:51:40.335673 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.335644    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa9b69c_543e_49dd_b4f9_64c293ea4255.slice/crio-2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019 WatchSource:0}: Error finding container 2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019: Status 404 returned error can't find the container with id 2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019
Nov 02 19:51:40.336164 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.336141    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ebbd01_9145_45e5_965d_bcffc2f28d24.slice/crio-11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea WatchSource:0}: Error finding container 11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea: Status 404 returned error can't find the container with id 11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea
Nov 02 19:51:40.362886 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.362857    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4z8z6"]
Nov 02 19:51:40.365610 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.365584    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b25ee3_21e4_4256_b9d2_728b54e4d525.slice/crio-4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a WatchSource:0}: Error finding container 4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a: Status 404 returned error can't find the container with id 4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a
Nov 02 19:51:40.500427 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.500253    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sk775"]
Nov 02 19:51:40.502812 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.502767    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16088935_ac22_423a_b525_b5051055e3dc.slice/crio-5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6 WatchSource:0}: Error finding container 5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6: Status 404 returned error can't find the container with id 5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6
Nov 02 19:51:40.507466 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.507446    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kst75"]
Nov 02 19:51:40.510483 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:40.510460    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9711f2a_a824_4a90_afa0_9c2bd6661f6a.slice/crio-0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825 WatchSource:0}: Error finding container 0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825: Status 404 returned error can't find the container with id 0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825
Nov 02 19:51:40.511807 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.511775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:40.511950 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.511934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:40.560574 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.560528    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\""
Nov 02 19:51:40.561043 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.560636    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:51:40.561043 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.560702    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\""
Nov 02 19:51:40.704722 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.704689    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4z8z6" event={"ID":"73b25ee3-21e4-4256-b9d2-728b54e4d525","Type":"ContainerStarted","Data":"4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a"}
Nov 02 19:51:40.705822 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.705776    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5" event={"ID":"8fa9b69c-543e-49dd-b4f9-64c293ea4255","Type":"ContainerStarted","Data":"2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019"}
Nov 02 19:51:40.706875 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.706854    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" event={"ID":"15ebbd01-9145-45e5-965d-bcffc2f28d24","Type":"ContainerStarted","Data":"11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea"}
Nov 02 19:51:40.707913 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.707893    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp" event={"ID":"a482e020-2470-473c-aaa6-dda8c37d6b57","Type":"ContainerStarted","Data":"6bab0e624f1886020f171f64a9e209f964da681f09d013b7da4301d8428f7d97"}
Nov 02 19:51:40.709306 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.709285    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wbhwp" event={"ID":"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc","Type":"ContainerStarted","Data":"f7a44c73afcba63ec37c96f5d7f0f4753c4e19a28a7518cb7f260907e095d908"}
Nov 02 19:51:40.709395 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.709321    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wbhwp" event={"ID":"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc","Type":"ContainerStarted","Data":"26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411"}
Nov 02 19:51:40.710541 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.710523    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kst75" event={"ID":"e9711f2a-a824-4a90-afa0-9c2bd6661f6a","Type":"ContainerStarted","Data":"0562605e28a3658fcb5d1d2e977e01dc2d2e1b4ca611cfb5049046473f9e7f43"}
Nov 02 19:51:40.710625 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.710552    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kst75" event={"ID":"e9711f2a-a824-4a90-afa0-9c2bd6661f6a","Type":"ContainerStarted","Data":"0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825"}
Nov 02 19:51:40.711650 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.711631    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-chn6b" event={"ID":"94b6384d-6d2b-416b-900d-30cfa9f6ca93","Type":"ContainerStarted","Data":"bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5"}
Nov 02 19:51:40.713019 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.712997    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sk775" event={"ID":"16088935-ac22-423a-b525-b5051055e3dc","Type":"ContainerStarted","Data":"e20788a8e3347bd2fc8270ae0005cc2bbdb46d5d10ddc804d47125ba698b118d"}
Nov 02 19:51:40.713110 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.713037    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sk775" event={"ID":"16088935-ac22-423a-b525-b5051055e3dc","Type":"ContainerStarted","Data":"5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6"}
Nov 02 19:51:40.714086 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.714066    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs" event={"ID":"d4169626-adde-4b36-85e9-11a8c5bb2626","Type":"ContainerStarted","Data":"39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320"}
Nov 02 19:51:40.715093 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:40.715075    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c" event={"ID":"c7f9f30c-31b6-4d93-b608-19cfad05e344","Type":"ContainerStarted","Data":"b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4"}
Nov 02 19:51:41.311645 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:41.311849 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:41.311849 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:41.311849 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:41.312027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:41.312027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:41.312027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:41.312027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.311954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:41.312027 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.312008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:41.312241 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312182515 +0000 UTC m=+36.250358929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:41.312300 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312290    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:41.312352 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312330    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312318098 +0000 UTC m=+36.250494502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:41.312413 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312395    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:41.312460 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312428    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312417384 +0000 UTC m=+36.250593794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:41.312513 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312494    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:41.312513 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312505    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:41.312609 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312536    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312525993 +0000 UTC m=+36.250702394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:41.312609 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312596    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:41.312609 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312608    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:41.312754 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312637    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312627033 +0000 UTC m=+36.250803445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:41.312754 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312695    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:41.312754 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312716734 +0000 UTC m=+36.250893139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:41.312941 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312799    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:41.312941 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.3128229 +0000 UTC m=+36.250999305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:41.312941 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312890    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:41.312941 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:43.312909774 +0000 UTC m=+36.251086179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:41.313098 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.312975    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:51:41.313098 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:41.313004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:52:13.312995004 +0000 UTC m=+66.251171418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:51:41.414041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.413079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:41.414041 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.413160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:41.422665 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.422596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:41.422889 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.422867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:41.429214 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.429186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:41.435340 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.435316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:41.487937 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.487683    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kst75" podStartSLOduration=438.487662788 podStartE2EDuration="7m18.487662788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:41.485678134 +0000 UTC m=+34.423854550" watchObservedRunningTime="2026-11-02 19:51:41.487662788 +0000 UTC m=+34.425839210"
Nov 02 19:51:41.592105 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.590309    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wbhwp" podStartSLOduration=438.590293866 podStartE2EDuration="7m18.590293866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:41.590142441 +0000 UTC m=+34.528318864" watchObservedRunningTime="2026-11-02 19:51:41.590293866 +0000 UTC m=+34.528470285"
Nov 02 19:51:41.697141 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.696338    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sk775" podStartSLOduration=438.696316073 podStartE2EDuration="7m18.696316073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:41.695486676 +0000 UTC m=+34.633663099" watchObservedRunningTime="2026-11-02 19:51:41.696316073 +0000 UTC m=+34.634492497"
Nov 02 19:51:41.815971 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.815939    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5cr4"]
Nov 02 19:51:41.849168 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:41.849086    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f2l4l"]
Nov 02 19:51:41.864964 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:41.864923    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b91d97_3140_4872_9e51_68cc0ad1de35.slice/crio-fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0 WatchSource:0}: Error finding container fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0: Status 404 returned error can't find the container with id fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0
Nov 02 19:51:42.728158 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:42.728030    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5cr4" event={"ID":"b64f0c97-c8f6-4e42-ac7d-32b5d6694968","Type":"ContainerStarted","Data":"e4019ce206f425394ba6dd0cf1f769ec82b58fa96c63593aadc3327b2593e0bf"}
Nov 02 19:51:42.737321 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:42.737271    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f2l4l" event={"ID":"e1b91d97-3140-4872-9e51-68cc0ad1de35","Type":"ContainerStarted","Data":"fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0"}
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346618    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.346653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346685    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346706    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.346665542 +0000 UTC m=+40.284841965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346718    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346727    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346750    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.346731445 +0000 UTC m=+40.284907877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346840    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346882    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346843    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346922    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:43.346958 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.346964    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347017    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.346771559 +0000 UTC m=+40.284947959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347046    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.347034632 +0000 UTC m=+40.285211041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347065    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.347056199 +0000 UTC m=+40.285232605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.347073538 +0000 UTC m=+40.285249954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.347091016 +0000 UTC m=+40.285267427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:43.347951 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:43.347117    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.347109459 +0000 UTC m=+40.285285936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:43.742796 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.742743    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs" event={"ID":"d4169626-adde-4b36-85e9-11a8c5bb2626","Type":"ContainerStarted","Data":"29d115e5b7e94a5b938e2c9ab24a8f1ad9c4c916731ae1b7ff0678a1f65db168"}
Nov 02 19:51:43.849151 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:43.848396    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs" podStartSLOduration=411.839196894 podStartE2EDuration="6m54.848376706s" totalImagesPullingTime="3.009179812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:43.848265415 +0000 UTC m=+36.786441841" watchObservedRunningTime="2026-11-02 19:51:43.848376706 +0000 UTC m=+36.786553132"
Nov 02 19:51:46.757618 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.757581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/0.log"
Nov 02 19:51:46.758155 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.757635    2579 generic.go:356] "Generic (PLEG): container finished" podID="15ebbd01-9145-45e5-965d-bcffc2f28d24" containerID="5738b0616a91e8618cd02df271c9d2653de3222d2bce3c66e38a3d07fa22d0de" exitCode=255
Nov 02 19:51:46.758155 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.757726    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" event={"ID":"15ebbd01-9145-45e5-965d-bcffc2f28d24","Type":"ContainerDied","Data":"5738b0616a91e8618cd02df271c9d2653de3222d2bce3c66e38a3d07fa22d0de"}
Nov 02 19:51:46.758155 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.758031    2579 scope.go:122] "RemoveContainer" containerID="5738b0616a91e8618cd02df271c9d2653de3222d2bce3c66e38a3d07fa22d0de"
Nov 02 19:51:46.762704 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.762672    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp" event={"ID":"a482e020-2470-473c-aaa6-dda8c37d6b57","Type":"ContainerStarted","Data":"c83c10876b6d255e07346d4f0f3194b2e028585711e780c601bdcf2d36ad0816"}
Nov 02 19:51:46.765959 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.765929    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-chn6b" event={"ID":"94b6384d-6d2b-416b-900d-30cfa9f6ca93","Type":"ContainerStarted","Data":"14395c867bcdde5ce49b217e5bf8f3b1a856e847709b7f640bfd90cf3713bb85"}
Nov 02 19:51:46.970809 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:46.970719    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp" podStartSLOduration=412.364734996 podStartE2EDuration="6m57.970699269s" totalImagesPullingTime="5.605964273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:46.969357061 +0000 UTC m=+39.907533483" watchObservedRunningTime="2026-11-02 19:51:46.970699269 +0000 UTC m=+39.908875693"
Nov 02 19:51:47.050328 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.050251    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-66zt6"]
Nov 02 19:51:47.068011 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.067913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.087410 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.087302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.087410 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.087347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf64z\" (UniqueName: \"kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.087697 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.087483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.087697 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.087561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.087697 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.087625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.109037 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.108958    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-chn6b" podStartSLOduration=412.520189193 podStartE2EDuration="6m58.108882284s" totalImagesPullingTime="5.588693091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:47.07784533 +0000 UTC m=+40.016021752" watchObservedRunningTime="2026-11-02 19:51:47.108882284 +0000 UTC m=+40.047058704"
Nov 02 19:51:47.109757 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.109735    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-66zt6"]
Nov 02 19:51:47.117416 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.117380    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:51:47.117416 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.117387    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\""
Nov 02 19:51:47.117416 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.117414    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:51:47.188927 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.188889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189118 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.189025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189118 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.189053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf64z\" (UniqueName: \"kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189118 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.189112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189293 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.189165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189293 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.189273    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:51:47.189398 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.189338    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.689318739 +0000 UTC m=+40.627495152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : configmap "kube-rbac-proxy" not found
Nov 02 19:51:47.189458 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.189403    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.189509 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.189492    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:47.189569 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.189525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:47.689514816 +0000 UTC m=+40.627691222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:47.190086 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.190061    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.251748 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.246298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf64z\" (UniqueName: \"kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.296065 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.295985    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:51:47.330620 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.330061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.351745 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.351716    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.390997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.391442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391638    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391656    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391715    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.391696921 +0000 UTC m=+48.329873340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:47.392439 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391846    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.39187638 +0000 UTC m=+48.330052795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391949    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.391982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.39197145 +0000 UTC m=+48.330147852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392036    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.39205755 +0000 UTC m=+48.330233969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392133    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.392123933 +0000 UTC m=+48.330300346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392215    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392249    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.392238781 +0000 UTC m=+48.330415196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392322    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.392341851 +0000 UTC m=+48.330518266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392362    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392372    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:47.393400 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.392402    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:55.392392966 +0000 UTC m=+48.330569383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:47.492396 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.492361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.492561 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.492462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.492561 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.492547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.492689 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.492592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.493256 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.493230    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.493371 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.493344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.493768 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.493739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.545928 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.545899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.605371 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.605288    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"]
Nov 02 19:51:47.620372 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.620337    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"]
Nov 02 19:51:47.620991 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.620968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:47.634590 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.634567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:47.657915 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.657129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:47.661490 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.661455    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"]
Nov 02 19:51:47.668809 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.668722    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"]
Nov 02 19:51:47.669673 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.669497    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:47.669673 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.669505    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 19:51:47.669673 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.669611    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-572ps\""
Nov 02 19:51:47.695179 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.695144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.695349 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.695259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8ls\" (UniqueName: \"kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls\") pod \"migrator-66df5d496b-mzsst\" (UID: \"1fe965ae-443d-44b8-9228-617718d5271c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:47.695349 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.695269    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:51:47.695349 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.695334    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:48.695315521 +0000 UTC m=+41.633491935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : configmap "kube-rbac-proxy" not found
Nov 02 19:51:47.695541 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.695359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s66\" (UniqueName: \"kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66\") pod \"migrator-66df5d496b-lll72\" (UID: \"acd788ff-7e64-424e-b072-c47867de3d73\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:47.695541 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.695451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:47.695676 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.695652    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:47.695746 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.695732    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:48.695710996 +0000 UTC m=+41.633887411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:47.771820 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.771762    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4z8z6" event={"ID":"73b25ee3-21e4-4256-b9d2-728b54e4d525","Type":"ContainerStarted","Data":"c2399eef4d963793e68f4dc2f05eea9369a751bc04092c9e36638ff2dd2a55d8"}
Nov 02 19:51:47.774601 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.774536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:47.775009 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.774992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/0.log"
Nov 02 19:51:47.775101 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.775026    2579 generic.go:356] "Generic (PLEG): container finished" podID="15ebbd01-9145-45e5-965d-bcffc2f28d24" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e" exitCode=255
Nov 02 19:51:47.775165 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.775145    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" event={"ID":"15ebbd01-9145-45e5-965d-bcffc2f28d24","Type":"ContainerDied","Data":"25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"}
Nov 02 19:51:47.775225 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.775194    2579 scope.go:122] "RemoveContainer" containerID="5738b0616a91e8618cd02df271c9d2653de3222d2bce3c66e38a3d07fa22d0de"
Nov 02 19:51:47.775604 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.775582    2579 scope.go:122] "RemoveContainer" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"
Nov 02 19:51:47.775799 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:47.775759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:51:47.777573 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.777552    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c" event={"ID":"c7f9f30c-31b6-4d93-b608-19cfad05e344","Type":"ContainerStarted","Data":"10ee70ba25f83684cd20a6f66d9e361949d174eedb9b9166ebd73ee5289ea5e3"}
Nov 02 19:51:47.798047 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.797387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8ls\" (UniqueName: \"kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls\") pod \"migrator-66df5d496b-mzsst\" (UID: \"1fe965ae-443d-44b8-9228-617718d5271c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:47.798047 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.797444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s66\" (UniqueName: \"kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66\") pod \"migrator-66df5d496b-lll72\" (UID: \"acd788ff-7e64-424e-b072-c47867de3d73\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:47.852756 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.852723    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8ls\" (UniqueName: \"kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls\") pod \"migrator-66df5d496b-mzsst\" (UID: \"1fe965ae-443d-44b8-9228-617718d5271c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:47.853056 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.853010    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s66\" (UniqueName: \"kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66\") pod \"migrator-66df5d496b-lll72\" (UID: \"acd788ff-7e64-424e-b072-c47867de3d73\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:47.936187 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.934332    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c" podStartSLOduration=412.429400796 podStartE2EDuration="6m58.934314134s" totalImagesPullingTime="6.504913338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:47.933825495 +0000 UTC m=+40.872001920" watchObservedRunningTime="2026-11-02 19:51:47.934314134 +0000 UTC m=+40.872490558"
Nov 02 19:51:47.947969 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.947934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:47.965956 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:47.965926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:48.145884 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.145753    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4z8z6" podStartSLOduration=2.680166511 podStartE2EDuration="9.145729877s" totalImagesPullingTime="6.465563366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:48.143990472 +0000 UTC m=+41.082166892" watchObservedRunningTime="2026-11-02 19:51:48.145729877 +0000 UTC m=+41.083906300"
Nov 02 19:51:48.243562 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.243529    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:51:48.248290 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:48.248253    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20809f26_f6f0_4feb_b1b0_3027441c2e7d.slice/crio-49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869 WatchSource:0}: Error finding container 49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869: Status 404 returned error can't find the container with id 49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869
Nov 02 19:51:48.347137 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.347105    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"]
Nov 02 19:51:48.351326 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:48.351295    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd788ff_7e64_424e_b072_c47867de3d73.slice/crio-7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9 WatchSource:0}: Error finding container 7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9: Status 404 returned error can't find the container with id 7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9
Nov 02 19:51:48.403733 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.403655    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"]
Nov 02 19:51:48.409374 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:48.409301    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe965ae_443d_44b8_9228_617718d5271c.slice/crio-be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d WatchSource:0}: Error finding container be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d: Status 404 returned error can't find the container with id be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d
Nov 02 19:51:48.710758 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.710722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:48.710943 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.710844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:48.710943 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:48.710908    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:48.711042 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:48.710974    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 19:51:48.711042 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:48.710989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:50.710968386 +0000 UTC m=+43.649144800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:48.711042 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:48.711035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:50.711018516 +0000 UTC m=+43.649194922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : configmap "kube-rbac-proxy" not found
Nov 02 19:51:48.782667 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.782581    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst" event={"ID":"1fe965ae-443d-44b8-9228-617718d5271c","Type":"ContainerStarted","Data":"be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d"}
Nov 02 19:51:48.783993 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.783960    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72" event={"ID":"acd788ff-7e64-424e-b072-c47867de3d73","Type":"ContainerStarted","Data":"7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9"}
Nov 02 19:51:48.785765 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.785735    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f2l4l" event={"ID":"e1b91d97-3140-4872-9e51-68cc0ad1de35","Type":"ContainerStarted","Data":"ca81b64d59ac5fcefc70dceb95fd9a7224f1c07e83a7efb84dd89a8ef355a472"}
Nov 02 19:51:48.785917 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.785872    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:48.787465 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.787435    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" event={"ID":"20809f26-f6f0-4feb-b1b0-3027441c2e7d","Type":"ContainerStarted","Data":"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"}
Nov 02 19:51:48.787573 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.787476    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" event={"ID":"20809f26-f6f0-4feb-b1b0-3027441c2e7d","Type":"ContainerStarted","Data":"49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869"}
Nov 02 19:51:48.789487 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.789454    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5" event={"ID":"8fa9b69c-543e-49dd-b4f9-64c293ea4255","Type":"ContainerStarted","Data":"c9065d16923fb7adb4290091f2347f0b24f0a6094e1625cc8a03d6a6255c3538"}
Nov 02 19:51:48.791388 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.791370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:48.792009 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.791995    2579 scope.go:122] "RemoveContainer" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"
Nov 02 19:51:48.792145 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:48.792123    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:51:48.998639 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.998532    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5" podStartSLOduration=410.453313831 podStartE2EDuration="6m57.998513937s" totalImagesPullingTime="7.545200106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:48.997720698 +0000 UTC m=+41.935897123" watchObservedRunningTime="2026-11-02 19:51:48.998513937 +0000 UTC m=+41.936690362"
Nov 02 19:51:48.999757 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:48.999711    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f2l4l" podStartSLOduration=34.225648443 podStartE2EDuration="40.999695664s" totalImagesPullingTime="6.774047221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:48.892925709 +0000 UTC m=+41.831102131" watchObservedRunningTime="2026-11-02 19:51:48.999695664 +0000 UTC m=+41.937872091"
Nov 02 19:51:49.212813 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.211748    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" podStartSLOduration=2.211733063 podStartE2EDuration="2.211733063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:49.210807223 +0000 UTC m=+42.148983643" watchObservedRunningTime="2026-11-02 19:51:49.211733063 +0000 UTC m=+42.149909485"
Nov 02 19:51:49.908344 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.908310    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-645bg"]
Nov 02 19:51:49.923234 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.923205    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:49.923390 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.923247    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:49.923390 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.923333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:49.923675 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.923652    2579 scope.go:122] "RemoveContainer" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"
Nov 02 19:51:49.924068 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:49.923916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:51:49.964547 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.964515    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-645bg"]
Nov 02 19:51:49.973136 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.972492    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 19:51:49.973136 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:49.972579    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f2nsm\""
Nov 02 19:51:50.009052 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.009016    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 19:51:50.009230 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.009014    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 19:51:50.009230 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.009078    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 19:51:50.024637 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.024604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.024857 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.024830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.025004 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.024976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrx5p\" (UniqueName: \"kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.126388 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.125694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrx5p\" (UniqueName: \"kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.126388 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.125844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.126388 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.125941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.126710 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.126687    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.137346 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.137267    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.178722 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.178691    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrx5p\" (UniqueName: \"kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.254224 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.254192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:50.733522 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.732925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:50.733522 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.733109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:50.733522 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:50.733212    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:50.733522 ip-10-0-133-26 kubenswrapper[2579]: E1102 19:51:50.733292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.73326692 +0000 UTC m=+47.671443326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:50.733522 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:50.733481    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:51.035960 ip-10-0-133-26 kubenswrapper[2579]: I1102 19:51:51.035841    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-645bg"]
Nov 02 19:51:51.037910 ip-10-0-133-26 kubenswrapper[2579]: W1102 19:51:51.037886    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baadb16_f6b2_418f_be63_a9998abeedef.slice/crio-1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f WatchSource:0}: Error finding container 1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f: Status 404 returned error can't find the container with id 1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f
Nov 02 19:51:51.075612 ip-10-0-133-26 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 19:51:51.089650 ip-10-0-133-26 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 19:51:51.089876 ip-10-0-133-26 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 19:51:51.089932 ip-10-0-133-26 systemd[1]: kubelet.service: Consumed 3.244s CPU time.
Nov 02 19:51:51.092437 ip-10-0-133-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 19:51:51.168870 ip-10-0-133-26 kubenswrapper[5099]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:51.168870 ip-10-0-133-26 kubenswrapper[5099]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 19:51:51.168870 ip-10-0-133-26 kubenswrapper[5099]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:51.168870 ip-10-0-133-26 kubenswrapper[5099]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 19:51:51.171060 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.170970    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.171060 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171060    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.171060 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171065    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171068    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171072    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171075    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171077    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171080    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171083    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171086    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171089    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171092    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171095    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171098    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171101    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171104    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171106    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171109    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171113    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171118    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171122    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171125    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.171172 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171128    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171131    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171133    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171136    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171139    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171141    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171144    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171147    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171150    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171152    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171155    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171158    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171162    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171165    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171167    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171170    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171173    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171175    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171178    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.171662 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171181    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171184    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171187    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171189    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171192    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171195    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171197    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171199    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171202    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171204    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171207    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171209    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171212    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171215    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171218    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171221    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171224    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171227    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171230    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171232    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.172162 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171235    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171237    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171241    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171243    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171246    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171249    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171251    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171254    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171257    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171259    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171262    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171264    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171267    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171270    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171278    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171281    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171284    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171286    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171289    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.172661 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171291    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171294    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171296    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171299    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171301    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171304    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171307    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171309    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171312    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171315    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171317    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171320    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171322    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171325    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171328    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171330    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171333    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171336    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171338    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.173143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171341    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171343    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171346    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171348    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171351    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171353    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.171357    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172216    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172224    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172232    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172235    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172239    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172243    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172246    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172249    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172252    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172256    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172259    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172262    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.173636 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172264    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172267    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172270    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172273    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172275    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172278    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172281    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172284    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172287    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172290    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172292    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172295    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172298    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172301    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172303    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172306    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172308    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172311    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172314    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172316    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.174146 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172319    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172321    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172324    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172327    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172330    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172332    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172335    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172337    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172340    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172343    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172346    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172348    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172351    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172353    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172356    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172359    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172362    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172365    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172367    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.174640 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172370    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172373    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172376    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172378    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172381    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172384    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172386    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172389    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172391    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172394    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172396    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172399    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172401    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172404    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172407    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172409    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172411    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172418    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172421    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.175181 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172423    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172426    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172428    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172432    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172436    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172439    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172443    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172446    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172448    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172451    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172453    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172456    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172458    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172461    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172463    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172467    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172469    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172472    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172474    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.175678 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172477    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172480    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172482    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172485    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172487    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172489    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172492    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172495    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172497    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172500    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172503    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172506    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172509    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172513    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172516    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172519    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.172521    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173125    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173132    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173136    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.176166 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173139    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173142    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173144    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173147    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173150    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173153    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173156    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173159    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173162    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173165    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173168    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173170    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173173    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173175    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173179    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173181    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173184    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173186    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173189    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.176689 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173192    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173194    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173197    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173199    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173201    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173210    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173213    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173216    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173218    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173224    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173227    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173229    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173232    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173234    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173237    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173239    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173242    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173245    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173248    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173253    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.177276 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173257    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173260    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173262    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173265    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173268    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173271    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173273    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173276    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173286    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173291    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173295    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173298    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173301    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173303    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173306    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173309    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173311    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173314    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173323    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.178051 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173326    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173328    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173331    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173334    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173337    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173339    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173342    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173344    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173347    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173350    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173353    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173356    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173359    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173361    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173364    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173367    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173369    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173372    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173374    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.178814 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173377    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173380    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173382    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173385    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173387    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173390    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173392    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173395    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173397    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173400    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173402    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173405    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173408    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173417    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173420    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173423    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173425    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173428    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173430    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173433    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.179611 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173436    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173438    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173441    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173443    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173446    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.173452    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173560    5099 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173567    5099 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173583    5099 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173588    5099 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173593    5099 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173596    5099 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173601    5099 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173605    5099 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173608    5099 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173611    5099 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173615    5099 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173618    5099 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173622    5099 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173627    5099 flags.go:64] FLAG: --cgroup-root=""
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173630    5099 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173634    5099 flags.go:64] FLAG: --client-ca-file=""
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173636    5099 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 19:51:51.180143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173640    5099 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173647    5099 flags.go:64] FLAG: --cluster-domain=""
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173650    5099 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173660    5099 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173664    5099 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173667    5099 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173671    5099 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173674    5099 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173678    5099 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173681    5099 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173684    5099 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173688    5099 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173691    5099 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173694    5099 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173698    5099 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173703    5099 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173706    5099 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173709    5099 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173712    5099 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173715    5099 flags.go:64] FLAG: --enable-server="true"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173718    5099 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173726    5099 flags.go:64] FLAG: --event-burst="100"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173729    5099 flags.go:64] FLAG: --event-qps="50"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173732    5099 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 19:51:51.180687 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173735    5099 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173738    5099 flags.go:64] FLAG: --eviction-hard=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173742    5099 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173745    5099 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173750    5099 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173754    5099 flags.go:64] FLAG: --eviction-soft=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173759    5099 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173763    5099 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173765    5099 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173768    5099 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173771    5099 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173775    5099 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173793    5099 flags.go:64] FLAG: --feature-gates=""
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173798    5099 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173802    5099 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173805    5099 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173808    5099 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173811    5099 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173815    5099 flags.go:64] FLAG: --help="false"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173818    5099 flags.go:64] FLAG: --hostname-override="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173821    5099 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173824    5099 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173829    5099 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 19:51:51.181301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173832    5099 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173836    5099 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173839    5099 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173842    5099 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173844    5099 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173847    5099 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173851    5099 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173854    5099 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173857    5099 flags.go:64] FLAG: --kube-reserved=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173860    5099 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173863    5099 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173866    5099 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173869    5099 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173873    5099 flags.go:64] FLAG: --lock-file=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173876    5099 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173880    5099 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173882    5099 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173888    5099 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173891    5099 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173894    5099 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173898    5099 flags.go:64] FLAG: --logging-format="text"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173901    5099 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173908    5099 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173911    5099 flags.go:64] FLAG: --manifest-url=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173914    5099 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 19:51:51.181906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173919    5099 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173922    5099 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173926    5099 flags.go:64] FLAG: --max-pods="110"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173929    5099 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173933    5099 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173936    5099 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173941    5099 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173944    5099 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173947    5099 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173950    5099 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173958    5099 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173961    5099 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173964    5099 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173968    5099 flags.go:64] FLAG: --pod-cidr=""
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173971    5099 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173974    5099 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173977    5099 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173979    5099 flags.go:64] FLAG: --port="10250"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173983    5099 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173985    5099 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0820451ca3f24c196"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173990    5099 flags.go:64] FLAG: --qos-reserved=""
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173993    5099 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173996    5099 flags.go:64] FLAG: --register-node="true"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.173999    5099 flags.go:64] FLAG: --register-with-taints=""
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174003    5099 flags.go:64] FLAG: --registry-burst="10"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174006    5099 flags.go:64] FLAG: --registry-qps="5"
Nov 02 19:51:51.182555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174008    5099 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174011    5099 flags.go:64] FLAG: --reserved-memory=""
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174015    5099 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174018    5099 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174027    5099 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174031    5099 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174034    5099 flags.go:64] FLAG: --runonce="false"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174037    5099 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174040    5099 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174043    5099 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174046    5099 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174050    5099 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174054    5099 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174058    5099 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174061    5099 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174064    5099 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174067    5099 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174070    5099 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174073    5099 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174076    5099 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174078    5099 flags.go:64] FLAG: --system-cgroups=""
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174081    5099 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174087    5099 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174089    5099 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174092    5099 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 19:51:51.183193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174099    5099 flags.go:64] FLAG: --tls-min-version=""
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174103    5099 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174106    5099 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174109    5099 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174112    5099 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174115    5099 flags.go:64] FLAG: --v="2"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174120    5099 flags.go:64] FLAG: --version="false"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174124    5099 flags.go:64] FLAG: --vmodule=""
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174129    5099 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174132    5099 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174290    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174294    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174298    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174301    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174303    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174306    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174309    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174311    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174314    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174318    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174320    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.183861 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174323    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174326    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174329    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174331    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174334    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174337    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174339    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174342    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174345    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174348    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174350    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174353    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174357    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174359    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174362    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174366    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174370    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174374    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174377    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.184400 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174380    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174382    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174385    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174388    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174390    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174393    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174396    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174399    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174401    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174404    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174406    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174411    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174413    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174416    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174418    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174421    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174423    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174426    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174428    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.185038 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174431    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174433    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174437    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174441    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174445    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174448    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174453    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174456    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174459    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174462    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174465    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174468    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174470    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174473    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174476    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174478    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174481    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174484    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174486    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.185596 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174490    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174492    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174495    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174498    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174500    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174505    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174508    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174510    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174513    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174516    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174518    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174521    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174523    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174527    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174529    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174532    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174534    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174537    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174540    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174545    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.186289 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174548    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174551    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174554    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174567    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174570    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174572    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174575    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174578    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174581    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174583    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174586    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174589    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174592    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174596    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174598    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174601    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174604    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.186888 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174607    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.174634    5099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174672    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174676    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174679    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174682    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174685    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174688    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174691    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174693    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174696    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174699    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174701    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174704    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174707    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174710    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.187373 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174713    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174715    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174719    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174723    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174726    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174730    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174732    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174735    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174737    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174740    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174743    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174745    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174748    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174751    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174754    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174757    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174761    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174763    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174766    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.187946 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174768    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174771    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174774    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174776    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174793    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174797    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174801    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174805    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174808    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174811    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174813    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174816    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174819    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174821    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174824    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174827    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174829    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174832    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174834    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174837    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.188635 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174839    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174842    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174845    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174848    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174850    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174853    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174855    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174859    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174861    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174866    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174869    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174871    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174875    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174878    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174880    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174883    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174885    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174888    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174890    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174893    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.189699 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174895    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174898    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174900    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174903    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174905    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174908    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174910    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174913    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174915    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174918    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174920    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174923    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174926    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174928    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174931    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174933    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174936    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174938    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174941    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.190534 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174943    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174946    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174950    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174953    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174955    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174960    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174964    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174966    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174969    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174972    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174975    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174977    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174980    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.174982    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.180119    5099 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.180142    5099 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.180166    5099 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.180172    5099 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180237    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.191329 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180242    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180246    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180249    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180252    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180255    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180258    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180261    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180264    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180267    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180269    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180272    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180288    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180292    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180296    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180298    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180301    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180304    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180307    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180310    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180313    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.192178 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180315    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180319    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180321    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180324    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180326    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180329    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180332    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180335    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180338    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180340    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180343    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180345    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180348    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180351    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180354    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180356    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180359    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180361    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180365    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180367    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.193026 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180370    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180372    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180375    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180378    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180380    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180383    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180386    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180389    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180391    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180394    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180397    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180400    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180403    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180406    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180408    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180411    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180414    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180416    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180419    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180422    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.193892 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180425    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180427    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180430    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180433    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180437    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180441    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180444    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180447    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180450    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180452    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180455    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180457    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180460    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180463    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180466    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180468    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180471    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180474    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180482    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.194737 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180485    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180488    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180490    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180493    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180496    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180499    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180501    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180504    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180506    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180510    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180514    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180518    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180521    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180524    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180527    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180529    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180532    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180535    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180537    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.195632 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180540    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180542    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180545    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180547    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180550    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180553    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180556    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.180585    5099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180728    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180733    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180736    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180739    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180742    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180751    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180755    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180758    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.196460 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180760    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180763    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180766    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180768    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180771    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180774    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180776    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180799    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180802    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180805    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180808    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180810    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180813    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180816    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180822    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180824    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180827    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180830    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180833    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.197152 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180836    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180839    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180841    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180844    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180847    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180850    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180852    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180855    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180858    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180860    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180863    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180867    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180870    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180872    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180875    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180878    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180880    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180883    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180885    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.198090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180890    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180894    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180897    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180901    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180903    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180906    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180909    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180911    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180916    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180918    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180921    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180924    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180927    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180929    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180932    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180935    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180940    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180944    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180948    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180952    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.198847 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180956    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180973    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180977    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180980    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180982    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180987    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180989    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180992    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180995    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.180998    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181000    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181003    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181006    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181008    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181011    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181014    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181016    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181019    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181022    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181025    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.199365 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181029    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181032    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181034    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181037    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181039    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181042    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181045    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181048    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181051    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181053    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181056    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181059    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181062    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181064    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181067    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181070    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181072    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181075    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181078    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.199874 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181081    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.181108    5099 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181145    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181149    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181153    5099 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181158    5099 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181162    5099 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181166    5099 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181169    5099 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181172    5099 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181174    5099 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181177    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181179    5099 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181182    5099 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181187    5099 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 19:51:51.200352 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181190    5099 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181192    5099 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181195    5099 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181198    5099 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181200    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181203    5099 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181206    5099 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181208    5099 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181211    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181214    5099 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181216    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181219    5099 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181222    5099 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181225    5099 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181227    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181230    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181232    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181236    5099 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181239    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181242    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 19:51:51.200794 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181244    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181247    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181250    5099 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181252    5099 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181255    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181258    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181260    5099 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181263    5099 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181266    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181268    5099 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181271    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181275    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181277    5099 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181287    5099 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181290    5099 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181293    5099 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181296    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181298    5099 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181301    5099 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181304    5099 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 19:51:51.201631 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181307    5099 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181310    5099 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181312    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181315    5099 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181318    5099 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181321    5099 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181323    5099 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181326    5099 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181329    5099 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181331    5099 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181336    5099 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181339    5099 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181341    5099 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181344    5099 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181347    5099 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181349    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181352    5099 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181354    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181357    5099 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 19:51:51.202464 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181359    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181362    5099 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181365    5099 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181368    5099 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181371    5099 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181374    5099 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181376    5099 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181378    5099 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181381    5099 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181384    5099 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181386    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181389    5099 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181391    5099 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181394    5099 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181398    5099 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181400    5099 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181403    5099 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181406    5099 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181408    5099 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 19:51:51.203057 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181411    5099 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181414    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181416    5099 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181419    5099 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181422    5099 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181425    5099 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181428    5099 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181430    5099 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181433    5099 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181436    5099 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181438    5099 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181441    5099 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181443    5099 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181448    5099 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:51.181450    5099 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.181522    5099 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182513    5099 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182573    5099 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182776    5099 server.go:1014] "Starting client certificate rotation"
Nov 02 19:51:51.203525 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182884    5099 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182943    5099 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:08 +0000 UTC" deadline="2028-04-03 16:17:32.534013853 +0000 UTC"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.182975    5099 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12428h25m41.351041732s"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.183304    5099 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.184443    5099 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.185723    5099 log.go:25] "Validated CRI v1 runtime API"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.187487    5099 log.go:25] "Validated CRI v1 image API"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.188234    5099 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 19:51:51.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.191304    5099 fs.go:135] Filesystem UUIDs: map[43ba860f-ca0f-42dd-a575-f4a2be5f8455:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a3b183f-ef26-486d-b3e1-16df02ff4c0c:/dev/nvme0n1p3]
Nov 02 19:51:51.204355 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.191333    5099 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bab0e624f1886020f171f64a9e209f964da681f09d013b7da4301d8428f7d97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bab0e624f1886020f171f64a9e209f964da681f09d013b7da4301d8428f7d97/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4019ce206f425394ba6dd0cf1f769ec82b58fa96c63593aadc3327b2593e0bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4019ce206f425394ba6dd0cf1f769ec82b58fa96c63593aadc3327b2593e0bf/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02d28c42-9ad6-4a4f-a070-19a056e3bd19/volumes/kubernetes.io~projected/kube-api-access-t8kw6:{mountpoint:/var/lib/kubelet/pods/02d28c42-9ad6-4a4f-a070-19a056e3bd19/volumes/kubernetes.io~projected/kube-api-access-t8kw6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/kube-api-access-blmlf:{mountpoint:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/kube-api-access-blmlf major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~projected/kube-api-access-x75r4:{mountpoint:/var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~projected/kube-api-access-x75r4 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16088935-ac22-423a-b525-b5051055e3dc/volumes/kubernetes.io~projected/kube-api-access-swnp7:{mountpoint:/var/lib/kubelet/pods/16088935-ac22-423a-b525-b5051055e3dc/volumes/kubernetes.io~projected/kube-api-access-swnp7 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/kube-api-access-sz28q:{mountpoint:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/kube-api-access-sz28q major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe965ae-443d-44b8-9228-617718d5271c/volumes/kubernetes.io~projected/kube-api-access-sm8ls:{mountpoint:/var/lib/kubelet/pods/1fe965ae-443d-44b8-9228-617718d5271c/volumes/kubernetes.io~projected/kube-api-access-sm8ls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20809f26-f6f0-4feb-b1b0-3027441c2e7d/volumes/kubernetes.io~projected/kube-api-access-pxs5l:{mountpoint:/var/lib/kubelet/pods/20809f26-f6f0-4feb-b1b0-3027441c2e7d/volumes/kubernetes.io~projected/kube-api-access-pxs5l major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26b9b178-30a3-4be9-ae42-663579a9bbab/volumes/kubernetes.io~projected/kube-api-access-p7whk:{mountpoint:/var/lib/kubelet/pods/26b9b178-30a3-4be9-ae42-663579a9bbab/volumes/kubernetes.io~projected/kube-api-access-p7whk major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45233480-479f-4e15-b887-48d7eae1a759/volumes/kubernetes.io~projected/kube-api-access-g27rl:{mountpoint:/var/lib/kubelet/pods/45233480-479f-4e15-b887-48d7eae1a759/volumes/kubernetes.io~projected/kube-api-access-g27rl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/490663ba-67e7-464e-ab56-9e5d2fdb1ca8/volumes/kubernetes.io~projected/kube-api-access-gjb54:{mountpoint:/var/lib/kubelet/pods/490663ba-67e7-464e-ab56-9e5d2fdb1ca8/volumes/kubernetes.io~projected/kube-api-access-gjb54 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/499f1915-5219-4094-9256-e01f23a3545b/volumes/kubernetes.io~projected/kube-api-access-z8kzc:{mountpoint:/var/lib/kubelet/pods/499f1915-5219-4094-9256-e01f23a3545b/volumes/kubernetes.io~projected/kube-api-access-z8kzc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d83e17a-0db2-48d2-9421-9926d7c11d52/volumes/kubernetes.io~projected/kube-api-access-bf64z:{mountpoint:/var/lib/kubelet/pods/5d83e17a-0db2-48d2-9421-9926d7c11d52/volumes/kubernetes.io~projected/kube-api-access-bf64z major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~projected/kube-api-access-cl4db:{mountpoint:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~projected/kube-api-access-cl4db major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~projected/kube-api-access-wrx5p:{mountpoint:/var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~projected/kube-api-access-wrx5p major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~projected/kube-api-access-zgwfv:{mountpoint:/var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~projected/kube-api-access-zgwfv major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654/volumes/kubernetes.io~projected/kube-api-access-t8vs8:{mountpoint:/var/lib/kubelet/pods/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654/volumes/kubernetes.io~projected/kube-api-access-t8vs8 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa9b69c-543e-49dd-b4f9-64c293ea4255/volumes/kubernetes.io~projected/kube-api-access-2c888:{mountpoint:/var/lib/kubelet/pods/8fa9b69c-543e-49dd-b4f9-64c293ea4255/volumes/kubernetes.io~projected/kube-api-access-2c888 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~projected/kube-api-access-wdh88:{mountpoint:/var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~projected/kube-api-access-wdh88 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~projected/kube-api-access-ptshk:{mountpoint:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~projected/kube-api-access-ptshk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~projected/kube-api-access-xd7cd:{mountpoint:/var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~projected/kube-api-access-xd7cd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acd788ff-7e64-424e-b072-c47867de3d73/volumes/kubernetes.io~projected/kube-api-access-b8s66:{mountpoint:/var/lib/kubelet/pods/acd788ff-7e64-424e-b072-c47867de3d73/volumes/kubernetes.io~projected/kube-api-access-b8s66 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b64f0c97-c8f6-4e42-ac7d-32b5d6694968/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b64f0c97-c8f6-4e42-ac7d-32b5d6694968/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~projected/kube-api-access-fwr48:{mountpoint:/var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~projected/kube-api-access-fwr48 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9d101f8-edc6-402a-a34c-922529a5b226/volumes/kubernetes.io~projected/kube-api-access-c5h77:{mountpoint:/var/lib/kubelet/pods/c9d101f8-edc6-402a-a34c-922529a5b226/volumes/kubernetes.io~projected/kube-api-access-c5h77 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cffe96fe-db7d-4362-bb90-25363f34503a/volumes/kubernetes.io~projected/kube-api-access-287d6:{mountpoint:/var/lib/kubelet/pods/cffe96fe-db7d-4362-bb90-25363f34503a/volumes/kubernetes.io~projected/kube-api-access-287d6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4169626-adde-4b36-85e9-11a8c5bb2626/volumes/kubernetes.io~projected/kube-api-access-gpstz:{mountpoint:/var/lib/kubelet/pods/d4169626-adde-4b36-85e9-11a8c5bb2626/volumes/kubernetes.io~projected/kube-api-access-gpstz major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1b91d97-3140-4872-9e51-68cc0ad1de35/volumes/kubernetes.io~projected/kube-api-access-frggk:{mountpoint:/var/lib/kubelet/pods/e1b91d97-3140-4872-9e51-68cc0ad1de35/volumes/kubernetes.io~projected/kube-api-access-frggk major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9711f2a-a824-4a90-afa0-9c2bd6661f6a/volumes/kubernetes.io~projected/kube-api-access-pln4f:{mountpoint:/var/lib/kubelet/pods/e9711f2a-a824-4a90-afa0-9c2bd6661f6a/volumes/kubernetes.io~projected/kube-api-access-pln4f major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc/volumes/kubernetes.io~projected/kube-api-access-qqscx:{mountpoint:/var/lib/kubelet/pods/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc/volumes/kubernetes.io~projected/kube-api-access-qqscx major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecde6df2-8fb8-4817-9314-a3d4a028ed6c/volumes/kubernetes.io~projected/kube-api-access-qt2q9:{mountpoint:/var/lib/kubelet/pods/ecde6df2-8fb8-4817-9314-a3d4a028ed6c/volumes/kubernetes.io~projected/kube-api-access-qt2q9 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f393cf71-bee3-472a-aa0f-e2be7e8c722a/volumes/kubernetes.io~projected/kube-api-access-2987g:{mountpoint:/var/lib/kubelet/pods/f393cf71-bee3-472a-aa0f-e2be7e8c722a/volumes/kubernetes.io~projected/kube-api-access-2987g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~projected/kube-api-access-wwkgn:{mountpoint:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~projected/kube-api-access-wwkgn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/19cd8b794de02374009c86314a01d6a024dfcafd2b5da4c414f2261b3896a153/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/af52eea814d3dcbe2693ae11261c1b003b1fcbe46c7f797db7c762d726c5f232/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ddfd4fa87683fe5fdfff0658ec07b6221848a23c3022e96471ce078ebeaa7458/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a40351bb449368cf3ebd6e6a39badf8c8186ed08e0301c25d1bff36207dde812/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6b80e76b946bc17d80a6022bc40f5f2b830875dbe0c6d604728ed19f6ea4e4a9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c9a9830eb0e4d19d453281d983ea1ce54fc011e356c25a44e26995ca5f1912a7/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0c777717bf91bfa678920aaa97cc580f7f2faed2f9bdbae35f441d5b13f5b7be/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/85809374351d3441344bf27724c4135ac35c40e956db101e171dbc8b82f688c9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a2e2f4fadcc5d7f030e0d8b1a3f661219ce49c2440e1ad2c783694a9bd2831ae/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b4334b91d3c5f6fcef72bb9fe3ace34485fda7f51443736c12a7a8c3a87b4c36/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/b77e8fb4fb532d14fd7a55347e148b88eb7dc4abdcefe7894b66ff72f0f897ee/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9349dad5f1507de554b8b1863f528572fee4661973fe2d303daa52c5fc6b3dc3/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4c0da98379ecd9ac3b134d721504ddd3726e55b392e8cfccad2a3929f15aa3d6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7b52364f7be469222d8daa98362f1474e9dba1cf5f1c3141671bf377516853ae/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5eca1b1ae28aa90b5bb9464be250d16db8b0204ca58aba964b155b09ee99e60d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dbab066b251202ad3ae649433f572fca3326036094f5362a8ee5a13e9fd98b52/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/132caf72647aae7e0431387e848c17cad788587cf687aa134ffcbaa102354eaf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/286eb300f95df9dcca642ccacea49fbe0146752c5e5cedf9ca3f9a2273a60206/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/27543a6a6742cbaa779b28e14e517e020852bc7d0006a080c21b7094b9ccbe47/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/e2aa4322278866033f2d071a2f182e3c5a3f0a45aebd23ef4c02bcf236b3b842/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-193:{mountpoint:/var/lib/containers/storage/overlay/2a753d22bbc8bdeb07303fd014fec1020ef6c8476e1c45ee1062d6dd958ce896/merged major:0 minor:193 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/975f81476a643af6efc4e4b26390cb9dfca382278be6dffce19f1a35900a12a9/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6a9f1f629801682de76144a8304a7cf38bba3dc9be3d0c38e795a4a01f874760/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8c7f518d167afb3055d2e307829a9d2c5d947867a0d240acf0d1ff6c81b00151/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/7cae8e9bf8e8745c4e7cecb46f25cd688c4f663db56894b10b3b1e16b7a8ed25/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c2290ee445f7c2b3353814182f4d6eaf8c59cadc8e193034423d79781b9a9cfb/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/025f356e7065432ec7ce05d39857bbc4cab6715f06c40442811cd9761a203660/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/fe9f0ea5f71c2b86afbf0c038af1aa4c2202b2e9d926e51397f9522c107dad59/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/6cbd0434784fb874157aa30c10d5702c575016ff0ce2cda141537104ef96add3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/89ccad8561e75102482669f2fcaecbafacfc429765d7884702e9443d6c6cc76f/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e102fafbc8bca623e50f24985fdb363c3ae46a883ad71e7b8b71609d48ceaa37/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a63a0ed290a4b10121aeda5a0afb77215444bd14f1e8cef726a5a13de0ebb2d1/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/7450bac8a597c6caa1e5f7fd2aeaeed2595e69babbf80b9f2ebe074c5adf2344/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/71d77f6d1a512f3ce1109ec5415511876e6032160cffa60fa0a06a9a1536485c/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/3b257a704023270f793f58a53a4b7427eea6a8994d32f4741f1cce9553ed6784/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/69b099bf5d655e190346db19c45ab10caaaeb714569a24fe7e309d6331c22d6a/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/92230c94fea4dea8ce09d9f3ca307e5949729dd3e3290ea89acb5416e5d465bf/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d0bdc18b4b4947c3d05e02c79bb2cca3b64f4e6911bd28d870a57ee34b939262/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/abc0d5557e5d8ecb6ad2d82c9b7a4e15c3e523b7658fa9183cbf488e961c085e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/3ef844994d0b5b4d5a9cb303c40adfce2cac53de42083f956b171497ec254d7c/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/68c5fcbd834679e04bc484584556c981e9e6fca7b79e82c53db74582bf7fa5d8/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/e6a3985233c34f20438fa70af6267560ca2a72122b40eb3edc453ea81709b582/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/9914e88fe94f11b38ba886ad3e2db2b292f789792d46296383bbae3cdb0db8fa/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/0021d0abc146cb9866fbad2ef0616a21411bb774d3be21b563b0dc4013bba9b6/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/d1fec0b94e57c0d147ef77067d6a52892abd0cef70315c41ff7a0ba844c0f6c7/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/700207aa7d41c9ff8a078c3bafa90dde6523b11fa0fc5685389707e0ed6b9932/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/7488aa177dc202df36e068ad544a049f1d47f7c4477f6b06178e343624b4538b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/292cdc8f47ab6aeb9245b35c57fdafa9f05611e438728a9f68e6e973ea03f86d/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9c1c35b8da2a6b937f5ee41dd4d344f681587b4ec75991a6727074d339de8544/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/25c4811df74dad4fa64bf90f47975db0aedd7362ee5a6c773e8e1daf970946fb/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/27e0f78b04c276f3cd44e366b3e6e9c57f87e32d52769b947708546e5c745d2c/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/65f6a8e1c40d75a238c7157ba8b3db5f8cce05d521939389366e2c98bbfe721f/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2dcf2f1a6a6b3a9eddb4b108b94122fc28836a0d522105c328f2f109a4867a9f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5871a121b91e9f1beb9bbdf37e544b5ca307e524b2a645844e12424aaf38a41a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dd138cbb9b267b4e51c14116849c69c190bba6260020066b89a1dc54d26bc8ee/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/21db31098c5432f488e5c34cc85806c1706ebda8b1d1cb84f7b76cfaa9290892/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a44d9d351cf71732af083906e203b27805d8ec3c9a6f01036589721715d49bad/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/58b0ae3e296ac4c2b5ade1aa23facc28c1d4679aea76f7817a4b5b1212db354c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b78d3d33cfc6841d2a76f31c04900efc47745bfba37335895e57d402c7d145ef/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7e8e961b9de585f9021a358fe29b0320ab157165eb12060c8ce48602de343797/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/333bbd5f345cc3eb3f0e717f68cd306a8838909f81c9ea8af2de6fe51936aa5f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/21956986de5c2376f354606f8ad53a6c7315bef02642013398469f4442666cc0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.203580    5099 manager.go:217] Machine: {Timestamp:2026-11-02 19:51:51.202994822 +0000 UTC m=+0.077265701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099965 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9c89391440ee743e96a699027915 SystemUUID:ec2f9c89-3914-40ee-743e-96a699027915 BootID:dbf57f0f-a209-470a-89a6-49c1b920b0f7 Filesystems:[{Device:/var/lib/kubelet/pods/45233480-479f-4e15-b887-48d7eae1a759/volumes/kubernetes.io~projected/kube-api-access-g27rl DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2960ae69b9c863680241aed86b69511f835dc001a71faff9cf9d48cacccaa5be/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f393cf71-bee3-472a-aa0f-e2be7e8c722a/volumes/kubernetes.io~projected/kube-api-access-2987g DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/490663ba-67e7-464e-ab56-9e5d2fdb1ca8/volumes/kubernetes.io~projected/kube-api-access-gjb54 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4628be7c96c26c62dae30701885db64dc3b4ee76ad3f08bf1b06472cfdf2e26a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26b9b178-30a3-4be9-ae42-663579a9bbab/volumes/kubernetes.io~projected/kube-api-access-p7whk DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/499f1915-5219-4094-9256-e01f23a3545b/volumes/kubernetes.io~projected/kube-api-access-z8kzc DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39b78cc5f18f4e6bc3d1f3361b5650513c6936eaaea36c1b9c704675bc08b320/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9cf72747cad301e41ee2d27410019ec772289cecac7dad2c7eccb35eab072f6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-193 DeviceMajor:0 DeviceMinor:193 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11ac07e3409c4499eb7e353eca729d06e0f1c970f417ec5633272087afb0ffea/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~projected/kube-api-access-xd7cd DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d83e17a-0db2-48d2-9421-9926d7c11d52/volumes/kubernetes.io~projected/kube-api-access-bf64z DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe965ae-443d-44b8-9228-617718d5271c/volumes/kubernetes.io~projected/kube-api-access-sm8ls DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4169626-adde-4b36-85e9-11a8c5bb2626/volumes/kubernetes.io~projected/kube-api-access-gpstz DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf0c67b0ca7c2bf21af514bf27e1f1e2dd4e4c1c56700b8d809916bd52ce97d5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20809f26-f6f0-4feb-b1b0-3027441c2e7d/volumes/kubernetes.io~projected/kube-api-access-pxs5l DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa9b69c-543e-49dd-b4f9-64c293ea4255/volumes/kubernetes.io~projected/kube-api-access-2c888 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c135898dfb094f5b29853a4908b83e618382b79f772ea8060dc25bb7f1fb019/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~projected/kube-api-access-wwkgn DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b48454c7ef03a8fe10a3ce641fb9987209e6581b66355a092c3b4b1f8cec295/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fcf032b1d18546f4283bbad48683760d8d247979f32307290bd62c48075111f/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bab0e624f1886020f171f64a9e209f964da681f09d013b7da4301d8428f7d97/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1b91d97-3140-4872-9e51-68cc0ad1de35/volumes/kubernetes.io~projected/kube-api-access-frggk DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~projected/kube-api-access-ptshk DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e468861ec5a73d0d1db99cfdc696aab0a5df59b77925a8d33eee7d3e0a7a0b9/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be9d74244d4920a1d3ef6303d0f212c8d0090eb5154f8eab33d95c623278885d/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbeed0e8-d128-4f10-b30c-f8930fc0837b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40c5360c67a9cb80c9bffd251b590f514403cfe195b3c7291808effad5140d79/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b64f0c97-c8f6-4e42-ac7d-32b5d6694968/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654/volumes/kubernetes.io~projected/kube-api-access-t8vs8 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~projected/kube-api-access-cl4db DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b00554ce85e4680d58623dff1ee1320ac0c5b7c1ea5464a936b838d30f0dcd6/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15ebbd01-9145-45e5-965d-bcffc2f28d24/volumes/kubernetes.io~projected/kube-api-access-x75r4 DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/kube-api-access-sz28q DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9711f2a-a824-4a90-afa0-9c2bd6661f6a/volumes/kubernetes.io~projected/kube-api-access-pln4f DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acd788ff-7e64-424e-b072-c47867de3d73/volumes/kubernetes.io~projected/kube-api-access-b8s66 DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ea072d8d4eb64f8dd3bd41fc4f1290c9755e919ccb39d15e6e9e167d79a8dbf/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/kube-api-access-blmlf DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7495a6cab068f361b021cd342616defaad757ad96238f3f6f8466c2133287202/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02d28c42-9ad6-4a4f-a070-19a056e3bd19/volumes/kubernetes.io~projected/kube-api-access-t8kw6 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9d101f8-edc6-402a-a34c-922529a5b226/volumes/kubernetes.io~projected/kube-api-access-c5h77 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46da4faae2704a03e40797eb3e6f2b28b5ed074e1326cdf77ec1dde136a92135/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~projected/kube-api-access-wdh88 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b009a7ea26b9b098d88b5df2f6e4da3430a12a04b4aadb766b6df81e7abd22c4/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4019ce206f425394ba6dd0cf1f769ec82b58fa96c63593aadc3327b2593e0bf/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4383d7-9852-4248-830b-1163ddc9a177/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecde6df2-8fb8-4817-9314-a3d4a028ed6c/volumes/kubernetes.io~projected/kube-api-access-qt2q9 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcddc35eb84a8226f8de1c064a4b7b554c50a68941f96c92e51cf081b6d004e0/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6baadb16-f6b2-418f-be63-a9998abeedef/volumes/kubernetes.io~projected/kube-api-access-wrx5p DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc/volumes/kubernetes.io~projected/kube-api-access-qqscx DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0335fd843325bf5db3f14cf3d4dc67b281ac232a57a2d66bb7e975f6eee2c825/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94b6384d-6d2b-416b-900d-30cfa9f6ca93/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a8bf33f-7796-49c0-a375-7932aa291af0/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a482e020-2470-473c-aaa6-dda8c37d6b57/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f9f30c-31b6-4d93-b608-19cfad05e344/volumes/kubernetes.io~projected/kube-api-access-fwr48 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16088935-ac22-423a-b525-b5051055e3dc/volumes/kubernetes.io~projected/kube-api-access-swnp7 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~projected/kube-api-access-zgwfv DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cffe96fe-db7d-4362-bb90-25363f34503a/volumes/kubernetes.io~projected/kube-api-access-287d6 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0be2204e7ad1c5ef0921a6e7fe2d8eb5b897e404f5abe62d56c0f92f1e706e1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73b25ee3-21e4-4256-b9d2-728b54e4d525/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26dca94267686390f2f39ffefd41c2f96cb7e613425aa38d2abb9a0fcd59c411/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0335fd843325bf5 MacAddress:82:2b:fa:41:91:53 Speed:10000 Mtu:8901} {Name:11ac07e3409c449 MacAddress:2a:e2:33:cc:11:dc Speed:10000 Mtu:8901} {Name:1fcf032b1d18546 MacAddress:1a:37:c8:e4:d4:7f Speed:10000 Mtu:8901} {Name:26dca9426768639 MacAddress:0a:d6:e1:cc:95:c5 Speed:10000 Mtu:8901} {Name:2c135898dfb094f MacAddress:42:8c:fe:9e:e9:8e Speed:10000 Mtu:8901} {Name:39b78cc5f18f4e6 MacAddress:3a:f2:d1:26:11:a3 Speed:10000 Mtu:8901} {Name:4628be7c96c26c6 MacAddress:86:c4:ee:30:33:10 Speed:10000 Mtu:8901} {Name:49e7d49c4dbd4cf MacAddress:52:b8:b4:1c:b1:bc Speed:10000 Mtu:8901} {Name:5b00554ce85e468 MacAddress:52:08:da:68:aa:b5 Speed:10000 Mtu:8901} {Name:6bab0e624f18860 MacAddress:b6:0a:66:ab:17:14 Speed:10000 Mtu:8901} {Name:7e468861ec5a73d MacAddress:76:51:a6:28:10:7f Speed:10000 Mtu:8901} {Name:b009a7ea26b9b09 MacAddress:0a:fa:24:b5:34:46 Speed:10000 Mtu:8901} {Name:be9d74244d4920a MacAddress:ae:7d:40:2c:22:02 Speed:10000 Mtu:8901} {Name:bf0c67b0ca7c2bf MacAddress:ea:8a:5b:d6:bb:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c7:f3:4f:5f Speed:0 Mtu:9001} {Name:br-int MacAddress:c2:c8:19:8e:61:db Speed:0 Mtu:8901} {Name:e4019ce206f4253 MacAddress:56:fc:00:8f:11:4f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c7:f3:4f:5f Speed:0 Mtu:9001} {Name:fcddc35eb84a822 MacAddress:9e:74:9b:23:0c:55 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:1d:4a:25:4b:3c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:7c:9d:33:30:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.203989    5099 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204040    5099 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204457    5099 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204497    5099 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204700    5099 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204711    5099 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204726    5099 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:51:51.204741 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204745    5099 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.204803    5099 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205046    5099 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205194    5099 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205242    5099 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205261    5099 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205277    5099 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205299    5099 kubelet.go:405] "Adding apiserver pod source"
Nov 02 19:51:51.205416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.205310    5099 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 19:51:51.206102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.206086    5099 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:51:51.206102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.206101    5099 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:51:51.206660 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.206642    5099 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 19:51:51.209006 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.208991    5099 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 19:51:51.209006 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209006    5099 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 19:51:51.209121 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209111    5099 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 19:51:51.209319 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209308    5099 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 19:51:51.209350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209338    5099 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 19:51:51.209433 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209424    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 19:51:51.209468 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209438    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 19:51:51.209468 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209446    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 19:51:51.209468 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209455    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 19:51:51.209468 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209463    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 19:51:51.209468 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209469    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209476    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209481    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209489    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209495    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209503    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209512    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209536    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 19:51:51.209602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.209547    5099 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 19:51:51.213115 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.213096    5099 server.go:1263] "Started kubelet"
Nov 02 19:51:51.213398 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.213352    5099 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 19:51:51.213638 ip-10-0-133-26 systemd[1]: Started Kubernetes Kubelet.
Nov 02 19:51:51.213936 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.213832    5099 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 19:51:51.213936 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.213934    5099 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 19:51:51.214254 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.214234    5099 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 19:51:51.214579 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.214568    5099 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 19:51:51.216486 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.216468    5099 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 19:51:51.216584 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.216499    5099 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 19:51:51.216584 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.216558    5099 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 19:51:51.216584 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.216577    5099 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 19:51:51.216584 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.216555    5099 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:09 +0000 UTC" deadline="2028-04-17 17:58:34.730640613 +0000 UTC"
Nov 02 19:51:51.218774 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.217339    5099 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12766h6m43.514052904s"
Nov 02 19:51:51.218774 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.217622    5099 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 19:51:51.218774 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.218402    5099 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221341    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221417    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ceb6f87-69c1-4d81-a0eb-e1b2702d9654" volumeName="kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir" seLinuxMountContext=""
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221435    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4169626-adde-4b36-85e9-11a8c5bb2626" volumeName="kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz" seLinuxMountContext=""
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221453    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeed0e8-d128-4f10-b30c-f8930fc0837b" volumeName="kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn" seLinuxMountContext=""
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221466    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d83e17a-0db2-48d2-9421-9926d7c11d52" volumeName="kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 19:51:51.221484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221478    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d83e17a-0db2-48d2-9421-9926d7c11d52" volumeName="kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221495    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acd788ff-7e64-424e-b072-c47867de3d73" volumeName="kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221509    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221529    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221541    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a8bf33f-7796-49c0-a375-7932aa291af0" volumeName="kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221557    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b25ee3-21e4-4256-b9d2-728b54e4d525" volumeName="kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221568    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a482e020-2470-473c-aaa6-dda8c37d6b57" volumeName="kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221580    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d101f8-edc6-402a-a34c-922529a5b226" volumeName="kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221598    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeed0e8-d128-4f10-b30c-f8930fc0837b" volumeName="kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221619    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeed0e8-d128-4f10-b30c-f8930fc0837b" volumeName="kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221637    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221649    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ebbd01-9145-45e5-965d-bcffc2f28d24" volumeName="kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221669    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221681    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="490663ba-67e7-464e-ab56-9e5d2fdb1ca8" volumeName="kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221699    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f9f30c-31b6-4d93-b608-19cfad05e344" volumeName="kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221710    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f938213c-c725-479c-a1aa-da876c4b9bc3" volumeName="kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221722    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b9b178-30a3-4be9-ae42-663579a9bbab" volumeName="kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221740    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a8bf33f-7796-49c0-a375-7932aa291af0" volumeName="kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221753    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6baadb16-f6b2-418f-be63-a9998abeedef" volumeName="kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221769    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221805    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a482e020-2470-473c-aaa6-dda8c37d6b57" volumeName="kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221818    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b64f0c97-c8f6-4e42-ac7d-32b5d6694968" volumeName="kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret" seLinuxMountContext=""
Nov 02 19:51:51.221836 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221836    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20809f26-f6f0-4feb-b1b0-3027441c2e7d" volumeName="kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 19:51:51.223078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221854    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20809f26-f6f0-4feb-b1b0-3027441c2e7d" volumeName="kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l" seLinuxMountContext=""
Nov 02 19:51:51.223078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.221863    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ceb6f87-69c1-4d81-a0eb-e1b2702d9654" volumeName="kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume" seLinuxMountContext=""
Nov 02 19:51:51.223078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.222351    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle" seLinuxMountContext=""
Nov 02 19:51:51.223078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.222407    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f9f30c-31b6-4d93-b608-19cfad05e344" volumeName="kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48" seLinuxMountContext=""
Nov 02 19:51:51.223078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.222425    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecde6df2-8fb8-4817-9314-a3d4a028ed6c" volumeName="kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9" seLinuxMountContext=""
Nov 02 19:51:51.223645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.222435    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f393cf71-bee3-472a-aa0f-e2be7e8c722a" volumeName="kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223655    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ebbd01-9145-45e5-965d-bcffc2f28d24" volumeName="kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223676    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d101f8-edc6-402a-a34c-922529a5b226" volumeName="kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223689    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223701    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6baadb16-f6b2-418f-be63-a9998abeedef" volumeName="kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223713    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223725    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223736    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f9f30c-31b6-4d93-b608-19cfad05e344" volumeName="kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config" seLinuxMountContext=""
Nov 02 19:51:51.223744 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223746    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeed0e8-d128-4f10-b30c-f8930fc0837b" volumeName="kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223758    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02d28c42-9ad6-4a4f-a070-19a056e3bd19" volumeName="kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223770    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223777    5099 factory.go:55] Registering systemd factory
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223801    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc" volumeName="kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223807    5099 factory.go:223] Registration of the systemd container factory successfully
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223814    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223828    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223839    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223850    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe965ae-443d-44b8-9228-617718d5271c" volumeName="kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223861    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="499f1915-5219-4094-9256-e01f23a3545b" volumeName="kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223893    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d83e17a-0db2-48d2-9421-9926d7c11d52" volumeName="kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223907    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a482e020-2470-473c-aaa6-dda8c37d6b57" volumeName="kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223921    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73b25ee3-21e4-4256-b9d2-728b54e4d525" volumeName="kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223948    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223959    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22ccf89-c8b1-46c3-a6ae-feaecb3a180f" volumeName="kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223967    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b22ccf89-c8b1-46c3-a6ae-feaecb3a180f" volumeName="kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.223990    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d101f8-edc6-402a-a34c-922529a5b226" volumeName="kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224001    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b91d97-3140-4872-9e51-68cc0ad1de35" volumeName="kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224013    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f393cf71-bee3-472a-aa0f-e2be7e8c722a" volumeName="kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224025    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02d28c42-9ad6-4a4f-a070-19a056e3bd19" volumeName="kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224038    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20809f26-f6f0-4feb-b1b0-3027441c2e7d" volumeName="kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224049    5099 factory.go:153] Registering CRI-O factory
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224062    5099 factory.go:223] Registration of the crio container factory successfully
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224112    5099 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224049    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45233480-479f-4e15-b887-48d7eae1a759" volumeName="kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224131    5099 factory.go:103] Registering Raw factory
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224132    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6baadb16-f6b2-418f-be63-a9998abeedef" volumeName="kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224148    5099 manager.go:1199] Started watching for new ooms in manager
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224145    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ceb6f87-69c1-4d81-a0eb-e1b2702d9654" volumeName="kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224337    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02d28c42-9ad6-4a4f-a070-19a056e3bd19" volumeName="kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224360    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224374    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ebbd01-9145-45e5-965d-bcffc2f28d24" volumeName="kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224386    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16088935-ac22-423a-b525-b5051055e3dc" volumeName="kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224395    5099 manager.go:319] Starting recovery of all containers
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224399    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa9b69c-543e-49dd-b4f9-64c293ea4255" volumeName="kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224413    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b6384d-6d2b-416b-900d-30cfa9f6ca93" volumeName="kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224425    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10b344ae-f1f6-4c55-bd15-11881859f44b" volumeName="kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224435    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ebbd01-9145-45e5-965d-bcffc2f28d24" volumeName="kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224446    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224458    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" volumeName="kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224468    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="499f1915-5219-4094-9256-e01f23a3545b" volumeName="kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224479    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4383d7-9852-4248-830b-1163ddc9a177" volumeName="kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224490    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbeed0e8-d128-4f10-b30c-f8930fc0837b" volumeName="kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224501    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20809f26-f6f0-4feb-b1b0-3027441c2e7d" volumeName="kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224548    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45233480-479f-4e15-b887-48d7eae1a759" volumeName="kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224562    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4383d7-9852-4248-830b-1163ddc9a177" volumeName="kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224575    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cffe96fe-db7d-4362-bb90-25363f34503a" volumeName="kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224587    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26b9b178-30a3-4be9-ae42-663579a9bbab" volumeName="kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224599    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a8bf33f-7796-49c0-a375-7932aa291af0" volumeName="kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224611    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4383d7-9852-4248-830b-1163ddc9a177" volumeName="kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224623    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9d101f8-edc6-402a-a34c-922529a5b226" volumeName="kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224634    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9711f2a-a824-4a90-afa0-9c2bd6661f6a" volumeName="kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224649    5099 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecde6df2-8fb8-4817-9314-a3d4a028ed6c" volumeName="kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config" seLinuxMountContext=""
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224660    5099 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 19:51:51.225358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.224666    5099 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 19:51:51.245681 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.245655    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:51.247289 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.247237    5099 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 19:51:51.249157 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.249132    5099 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 19:51:51.249276 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.249176    5099 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 19:51:51.249276 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.249205    5099 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 19:51:51.249395 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.249259    5099 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 19:51:51.256656 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.256634    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:51.270953 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.270924    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:51.271104 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.270965    5099 generic.go:356] "Generic (PLEG): container finished" podID="15ebbd01-9145-45e5-965d-bcffc2f28d24" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e" exitCode=255
Nov 02 19:51:51.283160 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.283120    5099 generic.go:356] "Generic (PLEG): container finished" podID="9bba4ed11659b02193d9121e647db325" containerID="51d0aedee076c669c46a8bfb27f3340de15b3e9d0d403bb29261cd0bf2cf0a70" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296290    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="d0c466670645cd6a70ca5f5bd7abd83a9e90c581859c4c84d60c4420befbd92d" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296315    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="4f2cd70c93e56697f4916257c56581b0bf129c61b5d5139585fef73f00d70c40" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296322    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="2b441afcd9cc47b1ebfcce515193243ee210143efc9ff90b032956c60d5ccd04" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296327    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="2421d86f4c0a3ed7cef161a77eed1653eecc494b07302d5590cf20a8065ea558" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296332    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="0130e176c1d0ae9c15393aa15cf5b082fa1fed4310eb5cc99811d155e44aefcd" exitCode=0
Nov 02 19:51:51.296354 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.296337    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9d101f8-edc6-402a-a34c-922529a5b226" containerID="68c31f946197bb6cf941081a83d7e8da5d7a2c85088f9a989ae94505b4a7b23b" exitCode=0
Nov 02 19:51:51.318482 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.318458    5099 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:51.326862 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.326835    5099 manager.go:324] Recovery completed
Nov 02 19:51:51.346605 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.346574    5099 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:51.347862 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.347842    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:51.347971 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.347876    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:51.347971 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.347891    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:51.349436 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.349417    5099 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 19:51:51.350902 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.350886    5099 cpu_manager.go:226] "Starting" policy="none"
Nov 02 19:51:51.350945 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.350902    5099 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:51.350983 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.350903    5099 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 19:51:51.351036 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351004    5099 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 19:51:51.351171 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351156    5099 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 19:51:51.351217 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351170    5099 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 19:51:51.351217 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351188    5099 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 19:51:51.351217 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351194    5099 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 19:51:51.351217 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351199    5099 policy_none.go:50] "Start"
Nov 02 19:51:51.351217 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351212    5099 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 19:51:51.351442 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351222    5099 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 19:51:51.351442 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351350    5099 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 19:51:51.351442 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351357    5099 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 19:51:51.351442 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351363    5099 policy_none.go:44] "Start"
Nov 02 19:51:51.351965 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351949    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:51.352020 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351979    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:51.352020 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.351992    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:51.356238 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.356212    5099 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 19:51:51.358747 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.358735    5099 manager.go:341] "Starting Device Plugin manager"
Nov 02 19:51:51.358864 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.358759    5099 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 19:51:51.358864 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.358771    5099 server.go:87] "Starting device plugin registration server"
Nov 02 19:51:51.359189 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359174    5099 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 19:51:51.359237 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359190    5099 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 19:51:51.359338 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359292    5099 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 19:51:51.359441 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359400    5099 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 19:51:51.359493 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359443    5099 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 19:51:51.359493 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359451    5099 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 19:51:51.359597 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.359506    5099 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:51:51.359432695Z","UUID":"95cd4217-947f-4cd8-bb75-19d812f01d32","Handler":null,"Name":"","Endpoint":""}
Nov 02 19:51:51.360842 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.360824    5099 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 19:51:51.360914 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.360850    5099 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 19:51:51.361653 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.361632    5099 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 19:51:51.361753 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.361676    5099 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-26.us-west-2.compute.internal\" not found"
Nov 02 19:51:51.403078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.403045    5099 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 19:51:51.403668 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.403648    5099 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:51.404761 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.404744    5099 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:51.404843 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.404746    5099 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:51.453855 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.453820    5099 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 19:51:51.459731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.459704    5099 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 19:51:51.460777 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.460761    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 19:51:51.460874 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.460806    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 19:51:51.460874 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.460817    5099 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 19:51:51.460874 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.460862    5099 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.550515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.550407    5099 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"]
Nov 02 19:51:51.551111 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551076    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"ee94c0e3fb310e2d7d2c04225780f5ac","Type":"ContainerStarted","Data":"7f1356af0cac5d0817c757036310db548c8f6a9c2ad9e0cafe52bbc8435b1091"}
Nov 02 19:51:51.551207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551116    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"ee94c0e3fb310e2d7d2c04225780f5ac","Type":"ContainerStarted","Data":"b293be6e2c09bcbf6a92c925ded000b84feb6e8bbcad3954989fd7edb4383597"}
Nov 02 19:51:51.551207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551130    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerStarted","Data":"fcb3944cee8d9af4b14052a0cb60080d40a140dceb9dd1fb02ef7df5bbca3d68"}
Nov 02 19:51:51.551207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551142    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerDied","Data":"51d0aedee076c669c46a8bfb27f3340de15b3e9d0d403bb29261cd0bf2cf0a70"}
Nov 02 19:51:51.551207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551154    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal" event={"ID":"9bba4ed11659b02193d9121e647db325","Type":"ContainerStarted","Data":"a9d3be29c6a4ff149f8de91f77926149d816480482461678fb28641d8e948224"}
Nov 02 19:51:51.551375 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551309    5099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.551482 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.551311    5099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.563243 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.563206    5099 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.563387 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.563298    5099 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.612205 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.612172    5099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:51.612347 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.612225    5099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.612347 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.612183    5099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:51.612347 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:51.612299    5099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.703966 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.703921    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.703966 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.703966    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.704157 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.703992    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804563    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804607    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804634    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804879 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804665    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804879 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804679    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee94c0e3fb310e2d7d2c04225780f5ac-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"ee94c0e3fb310e2d7d2c04225780f5ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:51.804879 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:51.804746    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bba4ed11659b02193d9121e647db325-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" (UID: \"9bba4ed11659b02193d9121e647db325\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:52.205732 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.205696    5099 apiserver.go:52] "Watching apiserver"
Nov 02 19:51:52.312874 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.312847    5099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:52.313040 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.312987    5099 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:52.351270 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.351240    5099 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 19:51:52.352143 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.352113    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v","openshift-image-registry/node-ca-s99qh","openshift-insights/periodic-gathering-s9fb4-64zpn","openshift-network-operator/iptables-alerter-zp64p","openshift-cluster-node-tuning-operator/tuned-b4m58","openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7","openshift-multus/network-metrics-daemon-rp4zx","openshift-console-operator/console-operator-7cd56b8dbc-s5vct","openshift-multus/multus-additional-cni-plugins-4drx6","openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5","openshift-network-console/networking-console-plugin-86c478bb88-6bkbd","kube-system/kas-connection-checker-685895c7c9-kst75","kube-system/global-pull-secret-syncer-g5cr4","kube-system/konnectivity-agent-szw22","kube-system/kas-connection-checker-685895c7c9-wbhwp","openshift-multus/multus-bv7hx","openshift-network-diagnostics/network-check-target-f2l4l","openshift-insights/insights-operator-7c4986d449-chn6b","openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst","openshift-service-ca/service-ca-5fffc97bdf-645bg","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs","openshift-image-registry/image-registry-6fb9fc8589-w7ktj","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp","kube-system/kas-connection-checker-685895c7c9-sk775","openshift-ingress-canary/ingress-canary-4z8z6","openshift-dns/node-resolver-4p96v","kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal","openshift-insights/insights-runtime-extractor-66zt6","openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72","openshift-ingress/router-default-ddb8948dc-fpq67","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c","openshift-image-registry/image-registry-85d85b8cd4-7btv7","openshift-dns/dns-default-wm7kv","openshift-ovn-kubernetes/ovnkube-node-hsmjq"]
Nov 02 19:51:52.355895 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.355868    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.356971 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.356948    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.358250 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.358222    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.359318 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.359302    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.360182 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.360165    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:52.363632 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.363612    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:52.366706 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.366685    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.367957 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.367940    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.368064 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.368047    5099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:52.368117 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.368103    5099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:52.368315 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.368299    5099 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 19:51:52.368389 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.368343    5099 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.us-west-2.compute.internal"
Nov 02 19:51:52.370046 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.370031    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.402068 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.402038    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.402068 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.402066    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.403176 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403157    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.403615 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403599    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.403873 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403858    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 19:51:52.403924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403872    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 19:51:52.403924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403910    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 19:51:52.403924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.403912    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k984j\""
Nov 02 19:51:52.404426 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.404412    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 19:51:52.404455 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.404447    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.404484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.404467    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-45fn9\""
Nov 02 19:51:52.404516 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.404496    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.404675 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.404658    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 19:51:52.415594 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.415569    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.420519 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.420496    5099 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 19:51:52.450608 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.450569    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 19:51:52.450608 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.450591    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f686f\""
Nov 02 19:51:52.450842 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.450594    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 19:51:52.451270 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451245    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 19:51:52.451366 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451282    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.451366 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451301    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 19:51:52.451366 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451334    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gf92x\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451363    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451378    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451429    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4dvwg\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451441    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451372    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451439    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.451524 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451499    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451534    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451540    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451573    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451584    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451613    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451638    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451650    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451714    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451745    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451799    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451718    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451812    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451827    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451749    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451846    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xq6qx\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451778    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.451876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451260    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451903    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5szdl\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451771    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451909    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.451988    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hphz2\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452110    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nrwfl\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452164    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452211    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452362    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452510    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452528    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452530    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452546    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452582    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452583    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452597    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 19:51:52.452617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452606    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452629    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452665    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fnfjn\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452684    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452715    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452722    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452725    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452769    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452804    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452831    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452902    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452832    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452816    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452922    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452808    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452955    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452960    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.452987    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453009    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453012    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453065    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453095    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453127    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453141    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453147    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453157    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453150    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453167    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-572ps\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453143    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453180    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453204    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453211    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453221    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453216    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453234    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453253    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453342    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f2nsm\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453376    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 19:51:52.453394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453399    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453427    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453450    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453452    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453489    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453523    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453564    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453609    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453659    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453709    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453750    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453883    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453936    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.453984    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 19:51:52.454568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.454187    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 19:51:52.508900 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508823    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.508900 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508860    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.508900 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508884    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.509103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508909    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s66\" (UniqueName: \"kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66\") pod \"migrator-66df5d496b-lll72\" (UID: \"acd788ff-7e64-424e-b072-c47867de3d73\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:52.509103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508943    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.509103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.508986    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.509103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509020    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499f1915-5219-4094-9256-e01f23a3545b-tmp-dir\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.509213 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509113    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpstz\" (UniqueName: \"kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz\") pod \"volume-data-source-validator-58c9f5f46d-6btrs\" (UID: \"d4169626-adde-4b36-85e9-11a8c5bb2626\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:52.509213 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509162    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.509213 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509201    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.509350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509231    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.509350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509257    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.509350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509290    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.509350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509318    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.509350 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509324    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-trusted-ca\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509372    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509408    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509457    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkgn\" (UniqueName: \"kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509490    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509551    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509566    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6baadb16-f6b2-418f-be63-a9998abeedef-signing-cabundle\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509572    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-original-pull-secret\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.509641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509603    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509685    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287d6\" (UniqueName: \"kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509715    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-script-lib\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509726    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509752    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b6384d-6d2b-416b-900d-30cfa9f6ca93-serving-cert\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509798    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509844    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509885    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6baadb16-f6b2-418f-be63-a9998abeedef-signing-key\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509891    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509909    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2987g\" (UniqueName: \"kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509925    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509941    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.509962    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.510018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510001    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510020    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510049    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-tmp-dir\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510060    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510066    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a482e020-2470-473c-aaa6-dda8c37d6b57-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510105    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510137    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8ls\" (UniqueName: \"kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls\") pod \"migrator-66df5d496b-mzsst\" (UID: \"1fe965ae-443d-44b8-9228-617718d5271c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510164    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510171    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-config-volume\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510191    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510206    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510232    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-service-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510300    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510318    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510352    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510362    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510383    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510390    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovnkube-config\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510413    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510436    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-tmp\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510446    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.510478 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510477    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510514    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510563    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510592    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510629    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75r4\" (UniqueName: \"kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510663    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510666    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510682    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptshk\" (UniqueName: \"kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510736    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510839    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510879    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510897    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510919    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510940    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.510966    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511001    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511032    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f938213c-c725-479c-a1aa-da876c4b9bc3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511072    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2q9\" (UniqueName: \"kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511097    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511116    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511210    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pln4f\" (UniqueName: \"kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f\") pod \"kas-connection-checker-685895c7c9-kst75\" (UID: \"e9711f2a-a824-4a90-afa0-9c2bd6661f6a\") " pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511248    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kzc\" (UniqueName: \"kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511267    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511291    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.511430 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511424    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrx5p\" (UniqueName: \"kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511456    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c888\" (UniqueName: \"kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888\") pod \"network-check-source-6f78d59f99-xxjw5\" (UID: \"8fa9b69c-543e-49dd-b4f9-64c293ea4255\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511627    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511667    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511697    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511732    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511802    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-tmp\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511814    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511843    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511862    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511882    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/45233480-479f-4e15-b887-48d7eae1a759-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511976    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.511986    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512001    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512032    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512088    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-agent-certs\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512105    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512130    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26b9b178-30a3-4be9-ae42-663579a9bbab-iptables-alerter-script\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512158    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512204    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512240    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqscx\" (UniqueName: \"kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx\") pod \"kas-connection-checker-685895c7c9-wbhwp\" (UID: \"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc\") " pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512268    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512293    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512321    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512335    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b6384d-6d2b-416b-900d-30cfa9f6ca93-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512338    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512365    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512388    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr48\" (UniqueName: \"kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512407    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.512484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512425    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512520    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-default-certificate\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512601    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4db\" (UniqueName: \"kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512631    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512670    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512698    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512727    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbeed0e8-d128-4f10-b30c-f8930fc0837b-ovn-node-metrics-cert\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512814    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512838    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512856    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512880    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kw6\" (UniqueName: \"kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512908    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.512951    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513003    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513050    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-stats-auth\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513072    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b22ccf89-c8b1-46c3-a6ae-feaecb3a180f-konnectivity-ca\") pod \"konnectivity-agent-szw22\" (UID: \"b22ccf89-c8b1-46c3-a6ae-feaecb3a180f\") " pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513083    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513138    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513161    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513175    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513177    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh88\" (UniqueName: \"kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513215    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513328    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513348    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513358    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cni-binary-copy\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513393    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513403    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7whk\" (UniqueName: \"kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513439    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.513464 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513468    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513493    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513508    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a482e020-2470-473c-aaa6-dda8c37d6b57-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513538    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vs8\" (UniqueName: \"kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513602    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-daemon-config\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513614    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513667    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7f9f30c-31b6-4d93-b608-19cfad05e344-config\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513684    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513757    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513813    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7f9f30c-31b6-4d93-b608-19cfad05e344-serving-cert\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513816    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513844    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513873    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513909    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513907    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513957    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513967    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513982    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjb54\" (UniqueName: \"kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.513997    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514045    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514084    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514112    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwfv\" (UniqueName: \"kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514185    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514275    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514308    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514318    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f393cf71-bee3-472a-aa0f-e2be7e8c722a-serviceca\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514352    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514414    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d83e17a-0db2-48d2-9421-9926d7c11d52-data-volume\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514439    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514464    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514505    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.514537 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514541    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnp7\" (UniqueName: \"kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7\") pod \"kas-connection-checker-685895c7c9-sk775\" (UID: \"16088935-ac22-423a-b525-b5051055e3dc\") " pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514569    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fbeed0e8-d128-4f10-b30c-f8930fc0837b-env-overrides\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514572    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514575    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c4383d7-9852-4248-830b-1163ddc9a177-etc-tuned\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514661    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514692    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf64z\" (UniqueName: \"kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514726    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h77\" (UniqueName: \"kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514875    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514909    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514924    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514939    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.514979    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515009    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27rl\" (UniqueName: \"kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515040    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515061    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515057    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15ebbd01-9145-45e5-965d-bcffc2f28d24-config\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515077    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515077    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15ebbd01-9145-45e5-965d-bcffc2f28d24-serving-cert\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515115    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515121    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b25ee3-21e4-4256-b9d2-728b54e4d525-cert\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515148    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515176    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515200    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515211    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/94b6384d-6d2b-416b-900d-30cfa9f6ca93-snapshots\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515228    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515268    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515297    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515318    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515406    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515419    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515465    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9d101f8-edc6-402a-a34c-922529a5b226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515538    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.515840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.515565    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.565891 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.565846    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpstz\" (UniqueName: \"kubernetes.io/projected/d4169626-adde-4b36-85e9-11a8c5bb2626-kube-api-access-gpstz\") pod \"volume-data-source-validator-58c9f5f46d-6btrs\" (UID: \"d4169626-adde-4b36-85e9-11a8c5bb2626\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6btrs"
Nov 02 19:51:52.565891 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.565878    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2987g\" (UniqueName: \"kubernetes.io/projected/f393cf71-bee3-472a-aa0f-e2be7e8c722a-kube-api-access-2987g\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.566204 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.566187    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.566503 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.566482    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8ls\" (UniqueName: \"kubernetes.io/projected/1fe965ae-443d-44b8-9228-617718d5271c-kube-api-access-sm8ls\") pod \"migrator-66df5d496b-mzsst\" (UID: \"1fe965ae-443d-44b8-9228-617718d5271c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzsst"
Nov 02 19:51:52.566658 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.566639    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2q9\" (UniqueName: \"kubernetes.io/projected/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-kube-api-access-qt2q9\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.566870 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.566850    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pln4f\" (UniqueName: \"kubernetes.io/projected/e9711f2a-a824-4a90-afa0-9c2bd6661f6a-kube-api-access-pln4f\") pod \"kas-connection-checker-685895c7c9-kst75\" (UID: \"e9711f2a-a824-4a90-afa0-9c2bd6661f6a\") " pod="kube-system/kas-connection-checker-685895c7c9-kst75"
Nov 02 19:51:52.569240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.568290    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287d6\" (UniqueName: \"kubernetes.io/projected/cffe96fe-db7d-4362-bb90-25363f34503a-kube-api-access-287d6\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:52.569240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.568581    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptshk\" (UniqueName: \"kubernetes.io/projected/9c4383d7-9852-4248-830b-1163ddc9a177-kube-api-access-ptshk\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.569240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.569172    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s66\" (UniqueName: \"kubernetes.io/projected/acd788ff-7e64-424e-b072-c47867de3d73-kube-api-access-b8s66\") pod \"migrator-66df5d496b-lll72\" (UID: \"acd788ff-7e64-424e-b072-c47867de3d73\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lll72"
Nov 02 19:51:52.569240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.569228    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"periodic-gathering-s9fb4-64zpn\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") " pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:51:52.570612 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.569995    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4db\" (UniqueName: \"kubernetes.io/projected/6a8bf33f-7796-49c0-a375-7932aa291af0-kube-api-access-cl4db\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.570612 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.570033    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrx5p\" (UniqueName: \"kubernetes.io/projected/6baadb16-f6b2-418f-be63-a9998abeedef-kube-api-access-wrx5p\") pod \"service-ca-5fffc97bdf-645bg\" (UID: \"6baadb16-f6b2-418f-be63-a9998abeedef\") " pod="openshift-service-ca/service-ca-5fffc97bdf-645bg"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571233    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkgn\" (UniqueName: \"kubernetes.io/projected/fbeed0e8-d128-4f10-b30c-f8930fc0837b-kube-api-access-wwkgn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571437    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571483    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/a482e020-2470-473c-aaa6-dda8c37d6b57-kube-api-access-xd7cd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp\" (UID: \"a482e020-2470-473c-aaa6-dda8c37d6b57\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571575    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf64z\" (UniqueName: \"kubernetes.io/projected/5d83e17a-0db2-48d2-9421-9926d7c11d52-kube-api-access-bf64z\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571864    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwfv\" (UniqueName: \"kubernetes.io/projected/73b25ee3-21e4-4256-b9d2-728b54e4d525-kube-api-access-zgwfv\") pod \"ingress-canary-4z8z6\" (UID: \"73b25ee3-21e4-4256-b9d2-728b54e4d525\") " pod="openshift-ingress-canary/ingress-canary-4z8z6"
Nov 02 19:51:52.572001 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.571984    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c888\" (UniqueName: \"kubernetes.io/projected/8fa9b69c-543e-49dd-b4f9-64c293ea4255-kube-api-access-2c888\") pod \"network-check-source-6f78d59f99-xxjw5\" (UID: \"8fa9b69c-543e-49dd-b4f9-64c293ea4255\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xxjw5"
Nov 02 19:51:52.572362 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.572279    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7whk\" (UniqueName: \"kubernetes.io/projected/26b9b178-30a3-4be9-ae42-663579a9bbab-kube-api-access-p7whk\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.572802 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.572425    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnp7\" (UniqueName: \"kubernetes.io/projected/16088935-ac22-423a-b525-b5051055e3dc-kube-api-access-swnp7\") pod \"kas-connection-checker-685895c7c9-sk775\" (UID: \"16088935-ac22-423a-b525-b5051055e3dc\") " pod="kube-system/kas-connection-checker-685895c7c9-sk775"
Nov 02 19:51:52.572802 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.572563    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h77\" (UniqueName: \"kubernetes.io/projected/c9d101f8-edc6-402a-a34c-922529a5b226-kube-api-access-c5h77\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.572802 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.572638    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqscx\" (UniqueName: \"kubernetes.io/projected/ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc-kube-api-access-qqscx\") pod \"kas-connection-checker-685895c7c9-wbhwp\" (UID: \"ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc\") " pod="kube-system/kas-connection-checker-685895c7c9-wbhwp"
Nov 02 19:51:52.572979 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.572805    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kw6\" (UniqueName: \"kubernetes.io/projected/02d28c42-9ad6-4a4f-a070-19a056e3bd19-kube-api-access-t8kw6\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573218    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573337    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjb54\" (UniqueName: \"kubernetes.io/projected/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kube-api-access-gjb54\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573352    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vs8\" (UniqueName: \"kubernetes.io/projected/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-kube-api-access-t8vs8\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573402    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573633    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh88\" (UniqueName: \"kubernetes.io/projected/94b6384d-6d2b-416b-900d-30cfa9f6ca93-kube-api-access-wdh88\") pod \"insights-operator-7c4986d449-chn6b\" (UID: \"94b6384d-6d2b-416b-900d-30cfa9f6ca93\") " pod="openshift-insights/insights-operator-7c4986d449-chn6b"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573689    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr48\" (UniqueName: \"kubernetes.io/projected/c7f9f30c-31b6-4d93-b608-19cfad05e344-kube-api-access-fwr48\") pod \"service-ca-operator-544d749c8c-b9g6c\" (UID: \"c7f9f30c-31b6-4d93-b608-19cfad05e344\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c"
Nov 02 19:51:52.573832 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573740    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frggk\" (UniqueName: \"kubernetes.io/projected/e1b91d97-3140-4872-9e51-68cc0ad1de35-kube-api-access-frggk\") pod \"network-check-target-f2l4l\" (UID: \"e1b91d97-3140-4872-9e51-68cc0ad1de35\") " pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:52.574065 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573958    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75r4\" (UniqueName: \"kubernetes.io/projected/15ebbd01-9145-45e5-965d-bcffc2f28d24-kube-api-access-x75r4\") pod \"console-operator-7cd56b8dbc-s5vct\" (UID: \"15ebbd01-9145-45e5-965d-bcffc2f28d24\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:52.574065 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.573997    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kzc\" (UniqueName: \"kubernetes.io/projected/499f1915-5219-4094-9256-e01f23a3545b-kube-api-access-z8kzc\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.574065 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.574028    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27rl\" (UniqueName: \"kubernetes.io/projected/45233480-479f-4e15-b887-48d7eae1a759-kube-api-access-g27rl\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.616755 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616716    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616755 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616754    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616772    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616806    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616825    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616860    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-ovn\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616875    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-socket-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616880    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-slash\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616881    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-var-lib-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616888    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616920    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-multus\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616936    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5d83e17a-0db2-48d2-9421-9926d7c11d52-crio-socket\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616950    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.616975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616972    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.616988    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617007    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617035    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617052    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-socket-dir-parent\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617059    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617079    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26b9b178-30a3-4be9-ae42-663579a9bbab-host-slash\") pod \"iptables-alerter-zp64p\" (UID: \"26b9b178-30a3-4be9-ae42-663579a9bbab\") " pod="openshift-network-operator/iptables-alerter-zp64p"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-log-socket\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617076    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617103    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-conf-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617118    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617127    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-run\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617144    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617167    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617181    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617196    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617219    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617233    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-bin\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617249    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-cni-netd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617252    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.117229769 +0000 UTC m=+1.991500658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617293    5099 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617309    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617324    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617352    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.117338955 +0000 UTC m=+1.991609851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:52.617360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617354    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617383    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.117366921 +0000 UTC m=+1.991637808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617382    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617309    5099 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617499    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617459    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617532    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.117512413 +0000 UTC m=+1.991783292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617576    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617614    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617643    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617653    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617670    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617697    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-netns\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617698    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617726    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-system-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617734    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617736    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-kubelet\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617755    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f393cf71-bee3-472a-aa0f-e2be7e8c722a-host\") pod \"node-ca-s99qh\" (UID: \"f393cf71-bee3-472a-aa0f-e2be7e8c722a\") " pod="openshift-image-registry/node-ca-s99qh"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617776    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617805    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-netns\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617835    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617861    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-system-cni-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617887    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617887    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617913    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617932    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617947    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617962    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617981    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.617997    5099 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618009    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-multus-certs\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618020    5099 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.617998    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618033    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-etc-openvswitch\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618051    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.118035513 +0000 UTC m=+1.992306393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618052    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-registration-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618065    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/499f1915-5219-4094-9256-e01f23a3545b-hosts-file\") pod \"node-resolver-4p96v\" (UID: \"499f1915-5219-4094-9256-e01f23a3545b\") " pod="openshift-dns/node-resolver-4p96v"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618075    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.118066776 +0000 UTC m=+1.992337642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618073    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-sys\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618111    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618129    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618143    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618157    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618172    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618207    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618221    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-run-k8s-cni-cncf-io\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618234    5099 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618250    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d101f8-edc6-402a-a34c-922529a5b226-cnibin\") pod \"multus-additional-cni-plugins-4drx6\" (UID: \"c9d101f8-edc6-402a-a34c-922529a5b226\") " pod="openshift-multus/multus-additional-cni-plugins-4drx6"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618254    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618269    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.118257893 +0000 UTC m=+1.992528766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618283    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-lib-modules\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618295    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-host\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.618286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618295    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618323    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-node-log\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618334    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618368    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618394    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618401    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysconfig\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618373    5099 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618420    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618431    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.118424699 +0000 UTC m=+1.992695565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618452    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618472    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618479    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-dbus\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618485    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618506    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-systemd-units\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618513    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-sysctl-conf\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618518    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618550    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618557    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618559    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-multus-cni-dir\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618575    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618598    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618618    5099 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618622    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-cnibin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618621    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.618653    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.118641302 +0000 UTC m=+1.992912196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618652    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-etc-kubernetes\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618651    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-modprobe-d\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618675    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618669    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-hostroot\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618693    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618710    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618735    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618759    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618766    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-device-dir\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618767    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-kubernetes\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618795    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-cni-bin\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618805    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-var-lib-kubelet\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618797    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618822    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/490663ba-67e7-464e-ab56-9e5d2fdb1ca8-sys-fs\") pod \"aws-ebs-csi-driver-node-h4zs8\" (UID: \"490663ba-67e7-464e-ab56-9e5d2fdb1ca8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4zs8"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618838    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c4383d7-9852-4248-830b-1163ddc9a177-etc-systemd\") pod \"tuned-b4m58\" (UID: \"9c4383d7-9852-4248-830b-1163ddc9a177\") " pod="openshift-cluster-node-tuning-operator/tuned-b4m58"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618842    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618876    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618887    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b64f0c97-c8f6-4e42-ac7d-32b5d6694968-kubelet-config\") pod \"global-pull-secret-syncer-g5cr4\" (UID: \"b64f0c97-c8f6-4e42-ac7d-32b5d6694968\") " pod="kube-system/global-pull-secret-syncer-g5cr4"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618900    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618930    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618943    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02d28c42-9ad6-4a4f-a070-19a056e3bd19-host-var-lib-kubelet\") pod \"multus-bv7hx\" (UID: \"02d28c42-9ad6-4a4f-a070-19a056e3bd19\") " pod="openshift-multus/multus-bv7hx"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.618962    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fbeed0e8-d128-4f10-b30c-f8930fc0837b-run-systemd\") pod \"ovnkube-node-hsmjq\" (UID: \"fbeed0e8-d128-4f10-b30c-f8930fc0837b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.619028    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.619039    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:52.619867 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:52.619078    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:53.119065957 +0000 UTC m=+1.993336838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:52.654171 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.654139    5099 scope.go:122] "RemoveContainer" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"
Nov 02 19:51:52.709005 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:52.708981    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s99qh_f393cf71-bee3-472a-aa0f-e2be7e8c722a/node-ca/0.log"
Nov 02 19:51:53.109412 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.109381    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4z8z6_73b25ee3-21e4-4256-b9d2-728b54e4d525/serve-healthcheck-canary/0.log"
Nov 02 19:51:53.124592 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.124555    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:53.124740 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.124607    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:53.124740 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.124630    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:53.124740 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.124658    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:53.124740 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124722    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:53.124740 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124738    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124745    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124749    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124758    5099 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.124774    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124808    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.124795735 +0000 UTC m=+2.999066614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124837    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.124821574 +0000 UTC m=+2.999092440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124861    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.124849862 +0000 UTC m=+2.999120734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124875    5099 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124878    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.124872323 +0000 UTC m=+2.999143189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.124938    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.124923316 +0000 UTC m=+2.999194185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.125000    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.125034    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.125075    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:53.125093 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125100    5099 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.125107    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125125    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.125119251 +0000 UTC m=+2.999390117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.125163    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125171    5099 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125192    5099 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125212    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.125195917 +0000 UTC m=+2.999466797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125220    5099 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125263    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.125252239 +0000 UTC m=+2.999523119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125265    5099 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125287    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.12527153 +0000 UTC m=+2.999542408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:53.125570 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.125306    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:54.125296444 +0000 UTC m=+2.999567313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:53.308933 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.308908    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lll72_acd788ff-7e64-424e-b072-c47867de3d73/migrator/0.log"
Nov 02 19:51:53.317541 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.317505    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-645bg" event={"ID":"6baadb16-f6b2-418f-be63-a9998abeedef","Type":"ContainerStarted","Data":"1623621873f79edb90861579cdae5b13fee0a1be8ba3c99b1ca35624a1fc2161"}
Nov 02 19:51:53.319057 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.319038    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 19:51:53.319466 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.319451    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/1.log"
Nov 02 19:51:53.319518 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.319481    5099 generic.go:356] "Generic (PLEG): container finished" podID="15ebbd01-9145-45e5-965d-bcffc2f28d24" containerID="04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133" exitCode=255
Nov 02 19:51:53.319586 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.319534    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" event={"ID":"15ebbd01-9145-45e5-965d-bcffc2f28d24","Type":"ContainerDied","Data":"04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133"}
Nov 02 19:51:53.319639 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.319589    5099 scope.go:122] "RemoveContainer" containerID="25207ada411a260f14f70fd4c7954da0be8344171b1483afc38ef3b3a43a9b5e"
Nov 02 19:51:53.320089 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.320075    5099 scope.go:122] "RemoveContainer" containerID="04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133"
Nov 02 19:51:53.320260 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:53.320232    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:51:53.509483 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.509400    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mzsst_1fe965ae-443d-44b8-9228-617718d5271c/migrator/0.log"
Nov 02 19:51:53.712956 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:53.712925    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp_a482e020-2470-473c-aaa6-dda8c37d6b57/kube-storage-version-migrator-operator/0.log"
Nov 02 19:51:54.134357 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134315    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:54.134540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134372    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:54.134540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134403    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:54.134540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134434    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:54.134540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134481    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:54.134540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134523    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:54.134823 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134547    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:54.134823 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134575    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:54.134823 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134602    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:54.134823 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.134650    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:54.134823 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.134805    5099 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:54.135056 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.134874    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.134855231 +0000 UTC m=+5.009126100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:54.135183 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135161    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:54.135269 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135183    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb9fc8589-w7ktj: secret "image-registry-tls" not found
Nov 02 19:51:54.135269 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135234    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls podName:1fd5af76-5b6f-4496-aae8-2cc745f64a2e nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135216348 +0000 UTC m=+5.009487214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls") pod "image-registry-6fb9fc8589-w7ktj" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e") : secret "image-registry-tls" not found
Nov 02 19:51:54.135373 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135280    5099 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 19:51:54.135373 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135310    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135293295 +0000 UTC m=+5.009564163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:54.135373 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135332    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert podName:f938213c-c725-479c-a1aa-da876c4b9bc3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135323338 +0000 UTC m=+5.009594211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bkbd" (UID: "f938213c-c725-479c-a1aa-da876c4b9bc3") : secret "networking-console-plugin-cert" not found
Nov 02 19:51:54.135373 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135334    5099 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:51:54.135373 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135368    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.13535942 +0000 UTC m=+5.009630287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135375    5099 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135408    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135397142 +0000 UTC m=+5.009668007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135434    5099 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135452    5099 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135467    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135457566 +0000 UTC m=+5.009728435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135482    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135472593 +0000 UTC m=+5.009743459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135524    5099 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135534    5099 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d85b8cd4-7btv7: secret "image-registry-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135537    5099 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135562    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135554933 +0000 UTC m=+5.009825799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:54.135620 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:54.135578    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls podName:10b344ae-f1f6-4c55-bd15-11881859f44b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:56.135569659 +0000 UTC m=+5.009840531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls") pod "image-registry-85d85b8cd4-7btv7" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b") : secret "image-registry-tls" not found
Nov 02 19:51:54.325634 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:54.325603    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 19:51:55.196954 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:55.196928    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:56.153077 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153034    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153179    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153214    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153238    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153339    5099 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153347    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153328905 +0000 UTC m=+9.027599789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : configmap references non-existent config key: service-ca.crt
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153390    5099 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153415    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls podName:45233480-479f-4e15-b887-48d7eae1a759 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.15340398 +0000 UTC m=+9.027674852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nfmz7" (UID: "45233480-479f-4e15-b887-48d7eae1a759") : secret "cluster-monitoring-operator-tls" not found
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153397    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153475    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153496    5099 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153505    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:56.153540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153544    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153559    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls podName:5d83e17a-0db2-48d2-9421-9926d7c11d52 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153540547 +0000 UTC m=+9.027811425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-66zt6" (UID: "5d83e17a-0db2-48d2-9421-9926d7c11d52") : secret "insights-runtime-extractor-tls" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153610    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153647    5099 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.153656    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153702    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153686395 +0000 UTC m=+9.027957286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153733    5099 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153766    5099 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153776    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153762844 +0000 UTC m=+9.028033742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153837    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153824033 +0000 UTC m=+9.028094914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153840    5099 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:51:56.154137 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:56.153882    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:52:00.153870577 +0000 UTC m=+9.028141459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:51:56.155769 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.155750    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"image-registry-85d85b8cd4-7btv7\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") " pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:56.155883 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.155834    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"image-registry-6fb9fc8589-w7ktj\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") " pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:56.155883 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.155843    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f938213c-c725-479c-a1aa-da876c4b9bc3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bkbd\" (UID: \"f938213c-c725-479c-a1aa-da876c4b9bc3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:56.258961 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.258931    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:56.261065 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.261036    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"
Nov 02 19:51:56.267827 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.267803    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:51:56.596327 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.596257    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:51:56.596517 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.596490    5099 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 19:51:56.724688 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.724656    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bkbd"]
Nov 02 19:51:56.727908 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.727874    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:51:56.763453 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:56.763425    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf938213c_c725_479c_a1aa_da876c4b9bc3.slice/crio-ec9dc5509ac69911143edb4e20b32f68f22c5d56b414d49b64dd1c45fdc16cfd WatchSource:0}: Error finding container ec9dc5509ac69911143edb4e20b32f68f22c5d56b414d49b64dd1c45fdc16cfd: Status 404 returned error can't find the container with id ec9dc5509ac69911143edb4e20b32f68f22c5d56b414d49b64dd1c45fdc16cfd
Nov 02 19:51:56.764086 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:56.764050    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b344ae_f1f6_4c55_bd15_11881859f44b.slice/crio-39ff1eeff377e07514fa000579043ec1d16ecd2c08540589669e884dd2dd4851 WatchSource:0}: Error finding container 39ff1eeff377e07514fa000579043ec1d16ecd2c08540589669e884dd2dd4851: Status 404 returned error can't find the container with id 39ff1eeff377e07514fa000579043ec1d16ecd2c08540589669e884dd2dd4851
Nov 02 19:51:56.768373 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.768352    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:51:56.774433 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:56.774390    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:51:56.776263 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:51:56.776237    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd5af76_5b6f_4496_aae8_2cc745f64a2e.slice/crio-ef5e6600a3964886f7798832e1fa6fd4d8c1b31c17a38e5c1dc162f8aad7d58c WatchSource:0}: Error finding container ef5e6600a3964886f7798832e1fa6fd4d8c1b31c17a38e5c1dc162f8aad7d58c: Status 404 returned error can't find the container with id ef5e6600a3964886f7798832e1fa6fd4d8c1b31c17a38e5c1dc162f8aad7d58c
Nov 02 19:51:57.021075 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.021042    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:51:57.348976 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.348886    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" event={"ID":"10b344ae-f1f6-4c55-bd15-11881859f44b","Type":"ContainerStarted","Data":"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"}
Nov 02 19:51:57.348976 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.348941    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" event={"ID":"10b344ae-f1f6-4c55-bd15-11881859f44b","Type":"ContainerStarted","Data":"39ff1eeff377e07514fa000579043ec1d16ecd2c08540589669e884dd2dd4851"}
Nov 02 19:51:57.349460 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.349056    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:51:57.350504 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.350476    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" event={"ID":"1fd5af76-5b6f-4496-aae8-2cc745f64a2e","Type":"ContainerStarted","Data":"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"}
Nov 02 19:51:57.350630 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.350514    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" event={"ID":"1fd5af76-5b6f-4496-aae8-2cc745f64a2e","Type":"ContainerStarted","Data":"ef5e6600a3964886f7798832e1fa6fd4d8c1b31c17a38e5c1dc162f8aad7d58c"}
Nov 02 19:51:57.351671 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.351650    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd" event={"ID":"f938213c-c725-479c-a1aa-da876c4b9bc3","Type":"ContainerStarted","Data":"ec9dc5509ac69911143edb4e20b32f68f22c5d56b414d49b64dd1c45fdc16cfd"}
Nov 02 19:51:57.940604 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.940561    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:51:57.941054 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:57.941029    5099 scope.go:122] "RemoveContainer" containerID="04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133"
Nov 02 19:51:57.941277 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:51:57.941257    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:51:58.357384 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:58.357351    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bkbd" event={"ID":"f938213c-c725-479c-a1aa-da876c4b9bc3","Type":"ContainerStarted","Data":"76e8afb4254b725dc16c93fb1679cede99e09705f21b1f643ad4ded136c94032"}
Nov 02 19:51:58.357806 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:51:58.357389    5099 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 19:52:00.191728 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191679    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:52:00.191728 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191733    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191761    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191819    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191847    5099 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191871    5099 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191891    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191914    5099 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191923    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls podName:ecde6df2-8fb8-4817-9314-a3d4a028ed6c nodeName:}" failed. No retries permitted until 2026-11-02 19:52:08.191902423 +0000 UTC m=+17.066173289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nzd7v" (UID: "ecde6df2-8fb8-4817-9314-a3d4a028ed6c") : secret "samples-operator-tls" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191952    5099 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.191970    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.191987    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls podName:8ceb6f87-69c1-4d81-a0eb-e1b2702d9654 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:08.191972408 +0000 UTC m=+17.066243280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls") pod "dns-default-wm7kv" (UID: "8ceb6f87-69c1-4d81-a0eb-e1b2702d9654") : secret "dns-default-metrics-tls" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.192008    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs podName:cffe96fe-db7d-4362-bb90-25363f34503a nodeName:}" failed. No retries permitted until 2026-11-02 19:52:08.191999339 +0000 UTC m=+17.066270209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs") pod "network-metrics-daemon-rp4zx" (UID: "cffe96fe-db7d-4362-bb90-25363f34503a") : secret "metrics-daemon-secret" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:00.192034    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs podName:6a8bf33f-7796-49c0-a375-7932aa291af0 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:08.192025804 +0000 UTC m=+17.066296671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs") pod "router-default-ddb8948dc-fpq67" (UID: "6a8bf33f-7796-49c0-a375-7932aa291af0") : secret "router-metrics-certs-default" not found
Nov 02 19:52:00.192184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.192070    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:52:00.192641 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.192604    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a8bf33f-7796-49c0-a375-7932aa291af0-service-ca-bundle\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:00.194238 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.194215    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5d83e17a-0db2-48d2-9421-9926d7c11d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-66zt6\" (UID: \"5d83e17a-0db2-48d2-9421-9926d7c11d52\") " pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:52:00.194376 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.194351    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/45233480-479f-4e15-b887-48d7eae1a759-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nfmz7\" (UID: \"45233480-479f-4e15-b887-48d7eae1a759\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:52:00.456355 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.456272    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"
Nov 02 19:52:00.470975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.470946    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-66zt6"
Nov 02 19:52:00.867981 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.867956    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7"]
Nov 02 19:52:00.868724 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:00.868697    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-66zt6"]
Nov 02 19:52:00.870090 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:00.870069    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d83e17a_0db2_48d2_9421_9926d7c11d52.slice/crio-a00d64554fbf5b9ed1e027776669a2ee6fe8a53af2fe84868d5ea47693970aee WatchSource:0}: Error finding container a00d64554fbf5b9ed1e027776669a2ee6fe8a53af2fe84868d5ea47693970aee: Status 404 returned error can't find the container with id a00d64554fbf5b9ed1e027776669a2ee6fe8a53af2fe84868d5ea47693970aee
Nov 02 19:52:00.870766 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:00.870743    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45233480_479f_4e15_b887_48d7eae1a759.slice/crio-4036122850aefdc22fc185d9433921efc68b5fc54c0431ec37d6eb39d3fc6abf WatchSource:0}: Error finding container 4036122850aefdc22fc185d9433921efc68b5fc54c0431ec37d6eb39d3fc6abf: Status 404 returned error can't find the container with id 4036122850aefdc22fc185d9433921efc68b5fc54c0431ec37d6eb39d3fc6abf
Nov 02 19:52:01.270336 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:01.270252    5099 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:52:01.270704 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:01.270657    5099 scope.go:122] "RemoveContainer" containerID="04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133"
Nov 02 19:52:01.270895 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:01.270876    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-s5vct_openshift-console-operator(15ebbd01-9145-45e5-965d-bcffc2f28d24)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24"
Nov 02 19:52:01.372637 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:01.372597    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66zt6" event={"ID":"5d83e17a-0db2-48d2-9421-9926d7c11d52","Type":"ContainerStarted","Data":"d631d552efc3090b425c23205b383bef680f1fe7e6a9f74af3a28a4e896d14b0"}
Nov 02 19:52:01.372840 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:01.372671    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66zt6" event={"ID":"5d83e17a-0db2-48d2-9421-9926d7c11d52","Type":"ContainerStarted","Data":"a00d64554fbf5b9ed1e027776669a2ee6fe8a53af2fe84868d5ea47693970aee"}
Nov 02 19:52:01.373859 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:01.373830    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7" event={"ID":"45233480-479f-4e15-b887-48d7eae1a759","Type":"ContainerStarted","Data":"4036122850aefdc22fc185d9433921efc68b5fc54c0431ec37d6eb39d3fc6abf"}
Nov 02 19:52:02.380382 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:02.380346    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66zt6" event={"ID":"5d83e17a-0db2-48d2-9421-9926d7c11d52","Type":"ContainerStarted","Data":"52821c12192dc798dcb767c2ac4ab0db4c3103883512487369d2e81478a78e86"}
Nov 02 19:52:02.446582 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:02.446545    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:52:03.387903 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:03.387865    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nfmz7" event={"ID":"45233480-479f-4e15-b887-48d7eae1a759","Type":"ContainerStarted","Data":"8aa30f0cbc66a390a5113e3d97b6b5ea18c962527220776a2869b1ed9f0d2175"}
Nov 02 19:52:04.249898 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.249868    5099 scope.go:122] "RemoveContainer" containerID="04941857a1fb3dfcd00d172bafa5391192440b0593037df2b8bf5bcfe0763133"
Nov 02 19:52:04.393215 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.393193    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 19:52:04.393648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.393305    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" event={"ID":"15ebbd01-9145-45e5-965d-bcffc2f28d24","Type":"ContainerStarted","Data":"4926b40c407872b8d18d5e306e5c08f93fe88b4c230b0496620c6c8fb4920905"}
Nov 02 19:52:04.393723 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.393660    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:52:04.395020 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.394996    5099 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-s5vct container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body=
Nov 02 19:52:04.395149 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.395047    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct" podUID="15ebbd01-9145-45e5-965d-bcffc2f28d24" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused"
Nov 02 19:52:04.395701 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:04.395669    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-66zt6" event={"ID":"5d83e17a-0db2-48d2-9421-9926d7c11d52","Type":"ContainerStarted","Data":"93a06caf880e0be4888b1a798419a37e9fe7c12b0505d7d4ce0f184cd27645d6"}
Nov 02 19:52:05.403628 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:05.403601    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-s5vct"
Nov 02 19:52:07.777297 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:07.777264    5099 patch_prober.go:29] interesting pod/image-registry-85d85b8cd4-7btv7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:07.777774 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:07.777331    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:08.265950 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.265907    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:52:08.266163 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.265961    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:52:08.266163 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.265993    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:08.266163 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.266022    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:08.268481 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.268454    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ceb6f87-69c1-4d81-a0eb-e1b2702d9654-metrics-tls\") pod \"dns-default-wm7kv\" (UID: \"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654\") " pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:08.268598 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.268570    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe96fe-db7d-4362-bb90-25363f34503a-metrics-certs\") pod \"network-metrics-daemon-rp4zx\" (UID: \"cffe96fe-db7d-4362-bb90-25363f34503a\") " pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:52:08.268705 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.268682    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecde6df2-8fb8-4817-9314-a3d4a028ed6c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nzd7v\" (UID: \"ecde6df2-8fb8-4817-9314-a3d4a028ed6c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:52:08.268740 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.268687    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8bf33f-7796-49c0-a375-7932aa291af0-metrics-certs\") pod \"router-default-ddb8948dc-fpq67\" (UID: \"6a8bf33f-7796-49c0-a375-7932aa291af0\") " pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:08.559005 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.558926    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:08.559994 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.559968    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"
Nov 02 19:52:08.564099 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.564078    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rp4zx"
Nov 02 19:52:08.568587 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:08.568560    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:09.036984 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.036947    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rp4zx"]
Nov 02 19:52:09.083538 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.083511    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wm7kv"]
Nov 02 19:52:09.084281 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.084253    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v"]
Nov 02 19:52:09.085097 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.085074    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddb8948dc-fpq67"]
Nov 02 19:52:09.085198 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:09.085094    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8bf33f_7796_49c0_a375_7932aa291af0.slice/crio-e8f96b686280ec6e45e305cafcf3351f1bc132aebd7f4e6d5b0ee15e4d69be79 WatchSource:0}: Error finding container e8f96b686280ec6e45e305cafcf3351f1bc132aebd7f4e6d5b0ee15e4d69be79: Status 404 returned error can't find the container with id e8f96b686280ec6e45e305cafcf3351f1bc132aebd7f4e6d5b0ee15e4d69be79
Nov 02 19:52:09.415599 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.415566    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddb8948dc-fpq67" event={"ID":"6a8bf33f-7796-49c0-a375-7932aa291af0","Type":"ContainerStarted","Data":"3e8229221004aa0ab5d2597c1102b9e2717abd275249e6f0d28f4a08f41f8b54"}
Nov 02 19:52:09.415844 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.415616    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddb8948dc-fpq67" event={"ID":"6a8bf33f-7796-49c0-a375-7932aa291af0","Type":"ContainerStarted","Data":"e8f96b686280ec6e45e305cafcf3351f1bc132aebd7f4e6d5b0ee15e4d69be79"}
Nov 02 19:52:09.416733 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.416708    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm7kv" event={"ID":"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654","Type":"ContainerStarted","Data":"47beecf14f3ba8490b24bded21a7b97e5ebbef7122a5a5182e0cdad985a5dfd1"}
Nov 02 19:52:09.417888 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.417865    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rp4zx" event={"ID":"cffe96fe-db7d-4362-bb90-25363f34503a","Type":"ContainerStarted","Data":"f3d7e08591026cec1ed00575713a448077322c233cbe5ab2889f37da2212c623"}
Nov 02 19:52:09.419429 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.419409    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v" event={"ID":"ecde6df2-8fb8-4817-9314-a3d4a028ed6c","Type":"ContainerStarted","Data":"207da6cc9f16b3754acc4fe50b9fbfa1bbcd4e1b0c273761bc4ced20b7784d19"}
Nov 02 19:52:09.559510 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.559470    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:09.562245 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.562219    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:09.814285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.814205    5099 patch_prober.go:29] interesting pod/image-registry-6fb9fc8589-w7ktj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:09.814285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:09.814254    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:10.139594 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.139555    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"]
Nov 02 19:52:10.143978 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.143520    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:10.144850 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.144629    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbf5j"]
Nov 02 19:52:10.148480 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.148101    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.158961 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.158937    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:52:10.184265 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.184233    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5937fd38-1cab-48bc-998d-d909080bad40-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp4hc\" (UID: \"5937fd38-1cab-48bc-998d-d909080bad40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:10.198769 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.198739    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 19:52:10.199011 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.198995    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d47jl\""
Nov 02 19:52:10.200570 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.200549    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 19:52:10.204061 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.204040    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 19:52:10.219593 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.219564    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"]
Nov 02 19:52:10.236154 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.236123    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-97bqd\""
Nov 02 19:52:10.240626 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.240600    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cc4c647b9-s52hf"]
Nov 02 19:52:10.244721 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.244696    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.264148 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.264117    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbf5j"]
Nov 02 19:52:10.285520 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285481    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-installation-pull-secrets\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285532    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszzb\" (UniqueName: \"kubernetes.io/projected/2dd35763-3166-4575-baab-fe8f5e343707-kube-api-access-mszzb\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285564    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-bound-sa-token\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285593    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-registry-certificates\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285621    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-registry-tls\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285651    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9e953c2-84ea-4256-be0a-0e75699409da-ca-trust-extracted\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285703 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285678    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd35763-3166-4575-baab-fe8f5e343707-tmp\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.285938 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285711    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5937fd38-1cab-48bc-998d-d909080bad40-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp4hc\" (UID: \"5937fd38-1cab-48bc-998d-d909080bad40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:10.285938 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285735    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-image-registry-private-configuration\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285938 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285764    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slw2l\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-kube-api-access-slw2l\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.285938 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.285819    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-trusted-ca\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.289240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.289212    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5937fd38-1cab-48bc-998d-d909080bad40-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qp4hc\" (UID: \"5937fd38-1cab-48bc-998d-d909080bad40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:10.304085 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.304050    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cc4c647b9-s52hf"]
Nov 02 19:52:10.387127 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387086    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd35763-3166-4575-baab-fe8f5e343707-tmp\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.387311 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387134    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-image-registry-private-configuration\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.388875 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387585    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd35763-3166-4575-baab-fe8f5e343707-tmp\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.388875 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387584    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw2l\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-kube-api-access-slw2l\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.388875 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387681    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-trusted-ca\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.388875 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.388801    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-trusted-ca\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.387772    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-installation-pull-secrets\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389217    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszzb\" (UniqueName: \"kubernetes.io/projected/2dd35763-3166-4575-baab-fe8f5e343707-kube-api-access-mszzb\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389248    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-bound-sa-token\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389282    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-registry-certificates\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389324    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-registry-tls\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389361    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9e953c2-84ea-4256-be0a-0e75699409da-ca-trust-extracted\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.389672    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9e953c2-84ea-4256-be0a-0e75699409da-ca-trust-extracted\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390490 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.390423    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9e953c2-84ea-4256-be0a-0e75699409da-registry-certificates\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.390900 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.390874    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-image-registry-private-configuration\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.392838 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.392813    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-registry-tls\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.393221 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.393179    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9e953c2-84ea-4256-be0a-0e75699409da-installation-pull-secrets\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.425091 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.425060    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:10.426613 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.426590    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddb8948dc-fpq67"
Nov 02 19:52:10.445115 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.445085    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slw2l\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-kube-api-access-slw2l\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.445451 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.445433    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9e953c2-84ea-4256-be0a-0e75699409da-bound-sa-token\") pod \"image-registry-6cc4c647b9-s52hf\" (UID: \"c9e953c2-84ea-4256-be0a-0e75699409da\") " pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.445734 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.445706    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszzb\" (UniqueName: \"kubernetes.io/projected/2dd35763-3166-4575-baab-fe8f5e343707-kube-api-access-mszzb\") pod \"downloads-6dd789c8b7-sbf5j\" (UID: \"2dd35763-3166-4575-baab-fe8f5e343707\") " pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.461974 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.461948    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:10.469929 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.469893    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:10.561324 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.561081    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:10.844255 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.844221    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"]
Nov 02 19:52:10.848730 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:10.848678    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5937fd38_1cab_48bc_998d_d909080bad40.slice/crio-2246b4012f65fdd651826e872213559d698d51b91fe17acfa5fd147d8fca20e6 WatchSource:0}: Error finding container 2246b4012f65fdd651826e872213559d698d51b91fe17acfa5fd147d8fca20e6: Status 404 returned error can't find the container with id 2246b4012f65fdd651826e872213559d698d51b91fe17acfa5fd147d8fca20e6
Nov 02 19:52:10.894889 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.894816    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sbf5j"]
Nov 02 19:52:10.899198 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:10.899161    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd35763_3166_4575_baab_fe8f5e343707.slice/crio-b72831941937e31b58599f82a39f95b8b759abdf7edbcc791eef725ad7db97e7 WatchSource:0}: Error finding container b72831941937e31b58599f82a39f95b8b759abdf7edbcc791eef725ad7db97e7: Status 404 returned error can't find the container with id b72831941937e31b58599f82a39f95b8b759abdf7edbcc791eef725ad7db97e7
Nov 02 19:52:10.956088 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:10.955759    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cc4c647b9-s52hf"]
Nov 02 19:52:10.958936 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:10.958895    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e953c2_84ea_4256_be0a_0e75699409da.slice/crio-a81664b0cc438ae699ab646b39e8c34ef143607ae3c0e39b19201bcc105c4641 WatchSource:0}: Error finding container a81664b0cc438ae699ab646b39e8c34ef143607ae3c0e39b19201bcc105c4641: Status 404 returned error can't find the container with id a81664b0cc438ae699ab646b39e8c34ef143607ae3c0e39b19201bcc105c4641
Nov 02 19:52:11.431199 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.431162    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc" event={"ID":"5937fd38-1cab-48bc-998d-d909080bad40","Type":"ContainerStarted","Data":"2246b4012f65fdd651826e872213559d698d51b91fe17acfa5fd147d8fca20e6"}
Nov 02 19:52:11.434408 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.434292    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf" event={"ID":"c9e953c2-84ea-4256-be0a-0e75699409da","Type":"ContainerStarted","Data":"e5de1103612557fa36ce304f78db57bf3bb0f2be77ea805e680dc0e87f342107"}
Nov 02 19:52:11.434408 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.434348    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf" event={"ID":"c9e953c2-84ea-4256-be0a-0e75699409da","Type":"ContainerStarted","Data":"a81664b0cc438ae699ab646b39e8c34ef143607ae3c0e39b19201bcc105c4641"}
Nov 02 19:52:11.434408 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.434375    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:11.438684 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.438654    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sbf5j" event={"ID":"2dd35763-3166-4575-baab-fe8f5e343707","Type":"ContainerStarted","Data":"b72831941937e31b58599f82a39f95b8b759abdf7edbcc791eef725ad7db97e7"}
Nov 02 19:52:11.440937 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.440901    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rp4zx" event={"ID":"cffe96fe-db7d-4362-bb90-25363f34503a","Type":"ContainerStarted","Data":"45b91f95d0489c6d3f7b9d3565b43ab958d1356f1ef93d3c44fc6a28532a9d67"}
Nov 02 19:52:11.441075 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.440966    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rp4zx" event={"ID":"cffe96fe-db7d-4362-bb90-25363f34503a","Type":"ContainerStarted","Data":"7a8611d1a7c324a8ba1d517a0b06588ccc32d97f93800c7f02fb548e75ec85ab"}
Nov 02 19:52:11.544770 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:11.544719    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf" podStartSLOduration=1.5447029319999999 podStartE2EDuration="1.544702932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:11.543990889 +0000 UTC m=+20.418261777" watchObservedRunningTime="2026-11-02 19:52:11.544702932 +0000 UTC m=+20.418973813"
Nov 02 19:52:12.452951 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:12.452591    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm7kv" event={"ID":"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654","Type":"ContainerStarted","Data":"c907da6e9a715dcc4de171a5b5f22007002e4b38be4e3a0eac8bcfb0b9a34e42"}
Nov 02 19:52:12.454846 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:12.454817    5099 patch_prober.go:29] interesting pod/image-registry-6fb9fc8589-w7ktj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:12.454964 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:12.454869    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:12.457429 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:12.457405    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v" event={"ID":"ecde6df2-8fb8-4817-9314-a3d4a028ed6c","Type":"ContainerStarted","Data":"1ceb3d591e0e0bb8a72c4b3f7c1518d326a9d7260e919a64b17cb101c00b7ba6"}
Nov 02 19:52:12.457472 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:12.457457    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nzd7v" event={"ID":"ecde6df2-8fb8-4817-9314-a3d4a028ed6c","Type":"ContainerStarted","Data":"0251589d1df63aa5bc5385294b717428e9ac0e03d91f198d9b9c4bb2d94505b0"}
Nov 02 19:52:13.464247 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.464213    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wm7kv" event={"ID":"8ceb6f87-69c1-4d81-a0eb-e1b2702d9654","Type":"ContainerStarted","Data":"c34e5a4b17d6ad7ab078f57668372a575d62f0ea5e01e5e7b5cf023f27c3e8d2"}
Nov 02 19:52:13.464729 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.464305    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:13.466038 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.466008    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc" event={"ID":"5937fd38-1cab-48bc-998d-d909080bad40","Type":"ContainerStarted","Data":"4adb7a49873188c8ae3d5ebef0bae496a1d614c7d00c4c43ec9a1ad461f04c31"}
Nov 02 19:52:13.466423 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.466401    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:13.472088 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.472064    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc"
Nov 02 19:52:13.678631 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:13.678570    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qp4hc" podStartSLOduration=1.817480209 podStartE2EDuration="3.678554967s" totalImagesPullingTime="1.861074758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:13.677958454 +0000 UTC m=+22.552229335" watchObservedRunningTime="2026-11-02 19:52:13.678554967 +0000 UTC m=+22.552825852"
Nov 02 19:52:14.234942 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:14.234908    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hsmjq"
Nov 02 19:52:16.099970 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:16.099930    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:52:19.812906 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:19.812863    5099 patch_prober.go:29] interesting pod/image-registry-6fb9fc8589-w7ktj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:19.813519 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:19.812931    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:20.167720 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:20.167679    5099 patch_prober.go:29] interesting pod/image-registry-85d85b8cd4-7btv7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:20.167972 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:20.167738    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:20.650263 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:20.650230    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szw22"
Nov 02 19:52:21.870459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:21.870419    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d8f7f"]
Nov 02 19:52:21.894141 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:21.894110    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005009 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.004976    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-metrics-client-ca\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005009 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005019    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-root\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005261 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005049    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-wtmp\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005261 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005130    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxp6\" (UniqueName: \"kubernetes.io/projected/4a3273d3-b9ac-479c-962f-98a89ffef16a-kube-api-access-rsxp6\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005261 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005162    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-tls\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005261 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005242    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005450 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005273    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-textfile\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005450 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005314    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-sys\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.005450 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.005371    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.050200 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.050162    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 19:52:22.050378 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.050218    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 19:52:22.050378 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.050286    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 19:52:22.050494 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.050379    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 19:52:22.050532 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.050497    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjzch\""
Nov 02 19:52:22.106284 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106247    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-metrics-client-ca\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106284 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106296    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-root\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106324    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-wtmp\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106355    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxp6\" (UniqueName: \"kubernetes.io/projected/4a3273d3-b9ac-479c-962f-98a89ffef16a-kube-api-access-rsxp6\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106383    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-tls\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106417    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106448    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-textfile\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106482    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-sys\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106561 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106503    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106804 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106705    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-wtmp\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.106804 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.106753    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-root\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.107125 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.107073    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-textfile\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.107227 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.107142    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a3273d3-b9ac-479c-962f-98a89ffef16a-sys\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.107286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.107229    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-metrics-client-ca\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.107329 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.107316    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.109080 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.109054    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.109203 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.109182    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a3273d3-b9ac-479c-962f-98a89ffef16a-node-exporter-tls\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.163724 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.163642    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxp6\" (UniqueName: \"kubernetes.io/projected/4a3273d3-b9ac-479c-962f-98a89ffef16a-kube-api-access-rsxp6\") pod \"node-exporter-d8f7f\" (UID: \"4a3273d3-b9ac-479c-962f-98a89ffef16a\") " pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.268175 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.268137    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8f7f"
Nov 02 19:52:22.300539 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:52:22.300507    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3273d3_b9ac_479c_962f_98a89ffef16a.slice/crio-89a8371b77a2829af0f74b9c9d38869ad10d83716872ae68f7536cd3ee6e7572 WatchSource:0}: Error finding container 89a8371b77a2829af0f74b9c9d38869ad10d83716872ae68f7536cd3ee6e7572: Status 404 returned error can't find the container with id 89a8371b77a2829af0f74b9c9d38869ad10d83716872ae68f7536cd3ee6e7572
Nov 02 19:52:22.453265 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.453166    5099 patch_prober.go:29] interesting pod/image-registry-6fb9fc8589-w7ktj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:22.453265 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.453230    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:22.512104 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:22.512066    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8f7f" event={"ID":"4a3273d3-b9ac-479c-962f-98a89ffef16a","Type":"ContainerStarted","Data":"89a8371b77a2829af0f74b9c9d38869ad10d83716872ae68f7536cd3ee6e7572"}
Nov 02 19:52:23.474512 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:23.474478    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wm7kv"
Nov 02 19:52:24.536702 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:24.536663    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8f7f" event={"ID":"4a3273d3-b9ac-479c-962f-98a89ffef16a","Type":"ContainerStarted","Data":"b83adbbdc82201ecdaa5f041324edc615e935c29076f1298e764177722512bfc"}
Nov 02 19:52:25.547446 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:25.547408    5099 generic.go:356] "Generic (PLEG): container finished" podID="4a3273d3-b9ac-479c-962f-98a89ffef16a" containerID="b83adbbdc82201ecdaa5f041324edc615e935c29076f1298e764177722512bfc" exitCode=0
Nov 02 19:52:25.547924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:25.547506    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8f7f" event={"ID":"4a3273d3-b9ac-479c-962f-98a89ffef16a","Type":"ContainerDied","Data":"b83adbbdc82201ecdaa5f041324edc615e935c29076f1298e764177722512bfc"}
Nov 02 19:52:25.547924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:25.547558    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8f7f" event={"ID":"4a3273d3-b9ac-479c-962f-98a89ffef16a","Type":"ContainerStarted","Data":"22ddccc65443fd4dfc3d5f3730f17f2719bc9007c40cd20b06d664c4300fc23a"}
Nov 02 19:52:25.547924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:25.547583    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8f7f" event={"ID":"4a3273d3-b9ac-479c-962f-98a89ffef16a","Type":"ContainerStarted","Data":"c3f67af93d650ce492bd9341dfe9b661cd747bd311f6c32befebb604188c9919"}
Nov 02 19:52:25.704366 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:25.704304    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d8f7f" podStartSLOduration=2.600780921 podStartE2EDuration="4.704288711s" totalImagesPullingTime="2.006436279s" totalInitContainerRuntime="97.071511ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:25.703841615 +0000 UTC m=+34.578112503" watchObservedRunningTime="2026-11-02 19:52:25.704288711 +0000 UTC m=+34.578559598"
Nov 02 19:52:27.024913 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:27.024883    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f2l4l"
Nov 02 19:52:29.576640 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:29.576595    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sbf5j" event={"ID":"2dd35763-3166-4575-baab-fe8f5e343707","Type":"ContainerStarted","Data":"dba3622ddfe5520d6c698dbdf5196cc957e6edfe9d2ba87424aa310e3767a153"}
Nov 02 19:52:29.577648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:29.577624    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:29.587492 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:29.587463    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sbf5j"
Nov 02 19:52:29.779590 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:29.779474    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-sbf5j" podStartSLOduration=2.179302513 podStartE2EDuration="19.779452514s" totalImagesPullingTime="17.600150001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:29.731833379 +0000 UTC m=+38.606104267" watchObservedRunningTime="2026-11-02 19:52:29.779452514 +0000 UTC m=+38.653723410"
Nov 02 19:52:30.165984 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:30.165953    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:52:30.567432 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:30.567357    5099 patch_prober.go:29] interesting pod/image-registry-6cc4c647b9-s52hf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 19:52:30.567432 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:30.567421    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf" podUID="c9e953c2-84ea-4256-be0a-0e75699409da" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:52:32.452352 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:32.452315    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:52:32.462750 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:32.462721    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cc4c647b9-s52hf"
Nov 02 19:52:32.750665 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:32.750587    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:52:35.187469 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.187420    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry" containerID="cri-o://cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0" gracePeriod=30
Nov 02 19:52:35.472928 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.472879    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:52:35.535466 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535418    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.535645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535482    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.535645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535538    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.535645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535565    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.535645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535620    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535689    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535721    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535748    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") pod \"10b344ae-f1f6-4c55-bd15-11881859f44b\" (UID: \"10b344ae-f1f6-4c55-bd15-11881859f44b\") "
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.535939    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.536043    5099 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-trusted-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.536515 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.536426    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:35.538885 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.538855    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:52:35.539154 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.539124    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:35.539540 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.539511    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:52:35.540719 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.540675    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "kube-api-access-blmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:35.540851 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.540765    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:35.546372 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.546344    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted" pod "10b344ae-f1f6-4c55-bd15-11881859f44b" (UID: "10b344ae-f1f6-4c55-bd15-11881859f44b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:35.606378 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.606328    5099 generic.go:356] "Generic (PLEG): container finished" podID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerID="cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0" exitCode=0
Nov 02 19:52:35.606552 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.606382    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7"
Nov 02 19:52:35.606552 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.606414    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" event={"ID":"10b344ae-f1f6-4c55-bd15-11881859f44b","Type":"ContainerDied","Data":"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"}
Nov 02 19:52:35.606552 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.606490    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d85b8cd4-7btv7" event={"ID":"10b344ae-f1f6-4c55-bd15-11881859f44b","Type":"ContainerDied","Data":"39ff1eeff377e07514fa000579043ec1d16ecd2c08540589669e884dd2dd4851"}
Nov 02 19:52:35.606552 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.606542    5099 scope.go:122] "RemoveContainer" containerID="cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"
Nov 02 19:52:35.617099 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.617078    5099 scope.go:122] "RemoveContainer" containerID="cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"
Nov 02 19:52:35.617435 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:35.617404    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0\": container with ID starting with cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0 not found: ID does not exist" containerID="cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"
Nov 02 19:52:35.617522 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.617442    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0"} err="failed to get container status \"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0\": rpc error: code = NotFound desc = could not find container \"cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0\": container with ID starting with cb2d4aa4692c7350b31be45930124b801f50f07fb559648599c14c0ee52e9bf0 not found: ID does not exist"
Nov 02 19:52:35.637565 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637534    5099 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-installation-pull-secrets\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637565 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637565    5099 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637821 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637581    5099 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10b344ae-f1f6-4c55-bd15-11881859f44b-image-registry-private-configuration\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637821 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637597    5099 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10b344ae-f1f6-4c55-bd15-11881859f44b-registry-certificates\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637821 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637613    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmlf\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-kube-api-access-blmlf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637821 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637628    5099 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10b344ae-f1f6-4c55-bd15-11881859f44b-bound-sa-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.637821 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.637641    5099 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10b344ae-f1f6-4c55-bd15-11881859f44b-ca-trust-extracted\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:35.781754 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.781674    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:52:35.784814 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:35.784770    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85d85b8cd4-7btv7"]
Nov 02 19:52:37.256577 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:37.256533    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" path="/var/lib/kubelet/pods/10b344ae-f1f6-4c55-bd15-11881859f44b/volumes"
Nov 02 19:52:43.299469 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:43.299425    5099 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-szw22_b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:44.414003 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:44.413965    5099 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-szw22_b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/konnectivity-agent/0.log" limit=1024
Nov 02 19:52:46.094642 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:46.094603    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:52:57.776087 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:57.776044    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry" containerID="cri-o://4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686" gracePeriod=30
Nov 02 19:52:58.046269 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.046203    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:52:58.140212 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140181    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140223    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140253    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140309    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140335    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140365    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140395    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.140645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140432    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") pod \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\" (UID: \"1fd5af76-5b6f-4496-aae8-2cc745f64a2e\") "
Nov 02 19:52:58.141131 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140822    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:58.141131 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.140839    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:52:58.142774 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.142731    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:52:58.143105 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.143014    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:58.143105 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.143075    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:52:58.143193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.143112    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:58.143193 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.143136    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "kube-api-access-sz28q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:52:58.159879 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.159839    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted" pod "1fd5af76-5b6f-4496-aae8-2cc745f64a2e" (UID: "1fd5af76-5b6f-4496-aae8-2cc745f64a2e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:52:58.241863 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241827    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz28q\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-kube-api-access-sz28q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.241863 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241856    5099 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-image-registry-private-configuration\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.241863 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241868    5099 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.242102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241878    5099 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-installation-pull-secrets\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.242102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241887    5099 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-trusted-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.242102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241898    5099 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-registry-certificates\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.242102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241907    5099 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-ca-trust-extracted\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.242102 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.241916    5099 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fd5af76-5b6f-4496-aae8-2cc745f64a2e-bound-sa-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:52:58.704697 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.704659    5099 generic.go:356] "Generic (PLEG): container finished" podID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerID="4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686" exitCode=0
Nov 02 19:52:58.704884 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.704706    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj"
Nov 02 19:52:58.704884 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.704750    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" event={"ID":"1fd5af76-5b6f-4496-aae8-2cc745f64a2e","Type":"ContainerDied","Data":"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"}
Nov 02 19:52:58.704884 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.704816    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb9fc8589-w7ktj" event={"ID":"1fd5af76-5b6f-4496-aae8-2cc745f64a2e","Type":"ContainerDied","Data":"ef5e6600a3964886f7798832e1fa6fd4d8c1b31c17a38e5c1dc162f8aad7d58c"}
Nov 02 19:52:58.704884 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.704840    5099 scope.go:122] "RemoveContainer" containerID="4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"
Nov 02 19:52:58.713700 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.713682    5099 scope.go:122] "RemoveContainer" containerID="4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"
Nov 02 19:52:58.713965 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:52:58.713947    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686\": container with ID starting with 4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686 not found: ID does not exist" containerID="4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"
Nov 02 19:52:58.714031 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.713977    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686"} err="failed to get container status \"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686\": rpc error: code = NotFound desc = could not find container \"4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686\": container with ID starting with 4b600efff817deeca1d04ef7e873249e52625eb0fd165b522fba6e35dfe57686 not found: ID does not exist"
Nov 02 19:52:58.911113 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.911074    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:52:58.958370 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:58.958290    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fb9fc8589-w7ktj"]
Nov 02 19:52:59.254992 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:52:59.254910    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" path="/var/lib/kubelet/pods/1fd5af76-5b6f-4496-aae8-2cc745f64a2e/volumes"
Nov 02 19:53:01.718225 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.718194    5099 generic.go:356] "Generic (PLEG): container finished" podID="a482e020-2470-473c-aaa6-dda8c37d6b57" containerID="c83c10876b6d255e07346d4f0f3194b2e028585711e780c601bdcf2d36ad0816" exitCode=0
Nov 02 19:53:01.718646 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.718261    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp" event={"ID":"a482e020-2470-473c-aaa6-dda8c37d6b57","Type":"ContainerDied","Data":"c83c10876b6d255e07346d4f0f3194b2e028585711e780c601bdcf2d36ad0816"}
Nov 02 19:53:01.718646 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.718609    5099 scope.go:122] "RemoveContainer" containerID="c83c10876b6d255e07346d4f0f3194b2e028585711e780c601bdcf2d36ad0816"
Nov 02 19:53:01.719864 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.719841    5099 generic.go:356] "Generic (PLEG): container finished" podID="94b6384d-6d2b-416b-900d-30cfa9f6ca93" containerID="14395c867bcdde5ce49b217e5bf8f3b1a856e847709b7f640bfd90cf3713bb85" exitCode=0
Nov 02 19:53:01.719942 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.719910    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-chn6b" event={"ID":"94b6384d-6d2b-416b-900d-30cfa9f6ca93","Type":"ContainerDied","Data":"14395c867bcdde5ce49b217e5bf8f3b1a856e847709b7f640bfd90cf3713bb85"}
Nov 02 19:53:01.720186 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:01.720167    5099 scope.go:122] "RemoveContainer" containerID="14395c867bcdde5ce49b217e5bf8f3b1a856e847709b7f640bfd90cf3713bb85"
Nov 02 19:53:02.725632 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:02.725594    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z2dhp" event={"ID":"a482e020-2470-473c-aaa6-dda8c37d6b57","Type":"ContainerStarted","Data":"25f747ebcfe4c1e87456aea7a527f403f3c049e0b49b23abf0de7342e9ebfb11"}
Nov 02 19:53:02.727371 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:02.727347    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-chn6b" event={"ID":"94b6384d-6d2b-416b-900d-30cfa9f6ca93","Type":"ContainerStarted","Data":"8c7e8317840bdcf718237f0587150a2a31f3ca2e2ca625087c0fca79b674aa45"}
Nov 02 19:53:02.746004 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:02.745972    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:53:02.746172 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:02.746156    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" podUID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" containerName="insights-gathering" containerID="cri-o://5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c" gracePeriod=30
Nov 02 19:53:03.033886 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.033856    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:53:03.181624 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.181587    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") pod \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") "
Nov 02 19:53:03.181824 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.181652    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") pod \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") "
Nov 02 19:53:03.181824 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.181680    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") pod \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") "
Nov 02 19:53:03.181824 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.181769    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") pod \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\" (UID: \"20809f26-f6f0-4feb-b1b0-3027441c2e7d\") "
Nov 02 19:53:03.182021 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.181950    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle" pod "20809f26-f6f0-4feb-b1b0-3027441c2e7d" (UID: "20809f26-f6f0-4feb-b1b0-3027441c2e7d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:53:03.182059 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.182002    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle" pod "20809f26-f6f0-4feb-b1b0-3027441c2e7d" (UID: "20809f26-f6f0-4feb-b1b0-3027441c2e7d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:53:03.182140 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.182116    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path" pod "20809f26-f6f0-4feb-b1b0-3027441c2e7d" (UID: "20809f26-f6f0-4feb-b1b0-3027441c2e7d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:53:03.184090 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.184061    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l" pod "20809f26-f6f0-4feb-b1b0-3027441c2e7d" (UID: "20809f26-f6f0-4feb-b1b0-3027441c2e7d"). InnerVolumeSpecName "kube-api-access-pxs5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:53:03.282572 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.282535    5099 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-trusted-ca-bundle\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:03.282572 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.282564    5099 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/20809f26-f6f0-4feb-b1b0-3027441c2e7d-archives-path\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:03.282572 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.282574    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxs5l\" (UniqueName: \"kubernetes.io/projected/20809f26-f6f0-4feb-b1b0-3027441c2e7d-kube-api-access-pxs5l\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:03.282801 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.282584    5099 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20809f26-f6f0-4feb-b1b0-3027441c2e7d-service-ca-bundle\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:53:03.732814 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.732767    5099 generic.go:356] "Generic (PLEG): container finished" podID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" containerID="5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c" exitCode=2
Nov 02 19:53:03.732814 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.732820    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s9fb4-64zpn"
Nov 02 19:53:03.733294 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.732842    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" event={"ID":"20809f26-f6f0-4feb-b1b0-3027441c2e7d","Type":"ContainerDied","Data":"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"}
Nov 02 19:53:03.733294 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.732883    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s9fb4-64zpn" event={"ID":"20809f26-f6f0-4feb-b1b0-3027441c2e7d","Type":"ContainerDied","Data":"49e7d49c4dbd4cf25eb556d426dde391c0b2f0023ae43ab69467149b4b253869"}
Nov 02 19:53:03.733294 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.732902    5099 scope.go:122] "RemoveContainer" containerID="5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"
Nov 02 19:53:03.740888 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.740867    5099 scope.go:122] "RemoveContainer" containerID="5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"
Nov 02 19:53:03.741168 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:53:03.741147    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c\": container with ID starting with 5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c not found: ID does not exist" containerID="5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"
Nov 02 19:53:03.741225 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.741178    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c"} err="failed to get container status \"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c\": rpc error: code = NotFound desc = could not find container \"5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c\": container with ID starting with 5ade96e6a1248189df311ee2a953ada69a2782d713e20f8599684b8e194cb84c not found: ID does not exist"
Nov 02 19:53:03.885769 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.885733    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:53:03.891310 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:03.891286    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-s9fb4-64zpn"]
Nov 02 19:53:05.255145 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:05.255113    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" path="/var/lib/kubelet/pods/20809f26-f6f0-4feb-b1b0-3027441c2e7d/volumes"
Nov 02 19:53:12.770658 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:12.770586    5099 generic.go:356] "Generic (PLEG): container finished" podID="c7f9f30c-31b6-4d93-b608-19cfad05e344" containerID="10ee70ba25f83684cd20a6f66d9e361949d174eedb9b9166ebd73ee5289ea5e3" exitCode=0
Nov 02 19:53:12.771032 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:12.770656    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c" event={"ID":"c7f9f30c-31b6-4d93-b608-19cfad05e344","Type":"ContainerDied","Data":"10ee70ba25f83684cd20a6f66d9e361949d174eedb9b9166ebd73ee5289ea5e3"}
Nov 02 19:53:12.771032 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:12.770969    5099 scope.go:122] "RemoveContainer" containerID="10ee70ba25f83684cd20a6f66d9e361949d174eedb9b9166ebd73ee5289ea5e3"
Nov 02 19:53:13.778294 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:53:13.778258    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-b9g6c" event={"ID":"c7f9f30c-31b6-4d93-b608-19cfad05e344","Type":"ContainerStarted","Data":"bb27bfea298aa7819a3915302921b839ff4688ecc4a2c7cce050620eb7275d25"}
Nov 02 19:53:16.092189 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:53:16.092153    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:53:46.092449 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:53:46.092402    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:54:16.092509 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:54:16.092433    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:54:46.093443 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:54:46.093404    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:55:16.092412 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:55:16.092375    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:55:46.092220 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:55:46.092134    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:56:16.092201 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:56:16.092163    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:56:41.033808 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.033751    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034114    5099 cpu_manager.go:414] "containerMap: removing container" podUID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" containerName="insights-gathering"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034128    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" containerName="insights-gathering"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034148    5099 cpu_manager.go:414] "containerMap: removing container" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034153    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034165    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034170    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034222    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="20809f26-f6f0-4feb-b1b0-3027441c2e7d" containerName="insights-gathering"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034232    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b344ae-f1f6-4c55-bd15-11881859f44b" containerName="registry"
Nov 02 19:56:41.034459 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.034239    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd5af76-5b6f-4496-aae8-2cc745f64a2e" containerName="registry"
Nov 02 19:56:41.037243 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.037225    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.083888 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.083851    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"]
Nov 02 19:56:41.086504 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.086482    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 19:56:41.087396 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.087376    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.105854 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.105824    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.105959 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.105930    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106000 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.105969    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106000 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.105988    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106068 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.106016    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106068 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.106056    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106138 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.106085    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106171 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.106156    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.106229 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.106215    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.130620 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.130592    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\""
Nov 02 19:56:41.131463 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.131437    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 19:56:41.140077 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.140054    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"]
Nov 02 19:56:41.186759 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.186722    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-66fnv\""
Nov 02 19:56:41.187078 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.187059    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 19:56:41.187124 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.187061    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 19:56:41.206801 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.206758    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.206960 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.206836    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.206960 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.206882    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207074 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207053    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207126 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207089    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207126 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207116    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.207232 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207159    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkcr\" (UniqueName: \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.207285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207259    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207336 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207323    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207348    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207394 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207375    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207492 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207407    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207492 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207463    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.207591 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207549    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207772 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207742    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.207950 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.207928    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.208176 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.208158    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.209267 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.209248    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.209595 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.209574    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.260472 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.260439    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.260653 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.260629    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\") pod \"maas-default-gateway-openshift-default-696f47c6c-jbvgn\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.308470 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.308389    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.308470 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.308426    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkcr\" (UniqueName: \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.308683 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.308555    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.308813 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.308771    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.308876 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.308851    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.318257 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.318231    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"]
Nov 02 19:56:41.323002 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.322982    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.349542 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.349516    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:41.362498 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.362470    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkcr\" (UniqueName: \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.374159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.374111    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-j24h5\""
Nov 02 19:56:41.376347 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.376317    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"]
Nov 02 19:56:41.403834 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.403723    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:56:41.409106 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409000    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409106 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409031    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w467h\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-kube-api-access-w467h\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409106 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409050    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409106 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409066    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409262 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409141    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409298 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409263    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409332 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409296    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cd8afc-945c-45c0-bff2-09e51914337c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409332 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409324    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.409419 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.409401    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cd8afc-945c-45c0-bff2-09e51914337c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.509889 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.509857    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510053 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.509914    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510129 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510046    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cd8afc-945c-45c0-bff2-09e51914337c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510129 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510100    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510231 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510194    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cd8afc-945c-45c0-bff2-09e51914337c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510284 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510266    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510342 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510295    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w467h\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-kube-api-access-w467h\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510342 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510308    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510342 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510325    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510493 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510370    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510493 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510473    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510658 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510640    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510728 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510688    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.510918 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.510898    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cd8afc-945c-45c0-bff2-09e51914337c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.512144 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.512123    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cd8afc-945c-45c0-bff2-09e51914337c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.512373 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.512357    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cd8afc-945c-45c0-bff2-09e51914337c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.564870 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.564771    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.565021 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.565000    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w467h\" (UniqueName: \"kubernetes.io/projected/70cd8afc-945c-45c0-bff2-09e51914337c-kube-api-access-w467h\") pod \"openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv\" (UID: \"70cd8afc-945c-45c0-bff2-09e51914337c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.636341 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.636302    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:41.762414 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.762384    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 19:56:41.764219 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:56:41.764193    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470b442b_bfe5_48dd_97e2_1b3581f33f28.slice/crio-e165c402ca731abeed48294cfa5681c60b0fcad0d81b24e4bd7f1dcc20dc7b2d WatchSource:0}: Error finding container e165c402ca731abeed48294cfa5681c60b0fcad0d81b24e4bd7f1dcc20dc7b2d: Status 404 returned error can't find the container with id e165c402ca731abeed48294cfa5681c60b0fcad0d81b24e4bd7f1dcc20dc7b2d
Nov 02 19:56:41.808674 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.808654    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"]
Nov 02 19:56:41.811227 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:56:41.811200    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c393fe3_4bad_466d_aa7f_737c43e4bdd1.slice/crio-668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9 WatchSource:0}: Error finding container 668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9: Status 404 returned error can't find the container with id 668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9
Nov 02 19:56:41.998662 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:41.998639    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"]
Nov 02 19:56:42.000522 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:56:42.000493    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cd8afc_945c_45c0_bff2_09e51914337c.slice/crio-f14010f756d249d8e6d94856dab28138fe8fbf58681066912ec8e1ded39006c3 WatchSource:0}: Error finding container f14010f756d249d8e6d94856dab28138fe8fbf58681066912ec8e1ded39006c3: Status 404 returned error can't find the container with id f14010f756d249d8e6d94856dab28138fe8fbf58681066912ec8e1ded39006c3
Nov 02 19:56:42.583398 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:42.583358    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv" event={"ID":"70cd8afc-945c-45c0-bff2-09e51914337c","Type":"ContainerStarted","Data":"f14010f756d249d8e6d94856dab28138fe8fbf58681066912ec8e1ded39006c3"}
Nov 02 19:56:42.584463 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:42.584439    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerStarted","Data":"668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9"}
Nov 02 19:56:42.585434 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:42.585407    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerStarted","Data":"e165c402ca731abeed48294cfa5681c60b0fcad0d81b24e4bd7f1dcc20dc7b2d"}
Nov 02 19:56:46.093842 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:56:46.093803    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:56:50.625016 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.624980    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerStarted","Data":"81f139d4b7ed5d36c94e4e4729ae09c75b4d5124840a6cf4ddceea85780da5fa"}
Nov 02 19:56:50.656205 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.656170    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:50.656338 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.656240    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:50.656338 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.656269    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:50.664178 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.664146    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:50.664286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.664210    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:50.664286 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:50.664243    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:56:51.285145 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.285113    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 19:56:51.288291 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.288266    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 19:56:51.633505 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.633458    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv" event={"ID":"70cd8afc-945c-45c0-bff2-09e51914337c","Type":"ContainerStarted","Data":"8710956837a188f55d705b6c145b40c59a7bfd5d82bba9b6cf140b36482c896b"}
Nov 02 19:56:51.636093 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.636065    5099 generic.go:356] "Generic (PLEG): container finished" podID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerID="81f139d4b7ed5d36c94e4e4729ae09c75b4d5124840a6cf4ddceea85780da5fa" exitCode=0
Nov 02 19:56:51.636220 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.636133    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerDied","Data":"81f139d4b7ed5d36c94e4e4729ae09c75b4d5124840a6cf4ddceea85780da5fa"}
Nov 02 19:56:51.636548 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.636525    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:51.637815 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.637773    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerStarted","Data":"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"}
Nov 02 19:56:51.642062 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.642040    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:51.840155 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.840102    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv" podStartSLOduration=2.187347699 podStartE2EDuration="10.84008597s" totalImagesPullingTime="8.652738271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:51.792220478 +0000 UTC m=+300.666491367" watchObservedRunningTime="2026-11-02 19:56:51.84008597 +0000 UTC m=+300.714356858"
Nov 02 19:56:51.902305 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:51.902208    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podStartSLOduration=3.004786204 podStartE2EDuration="11.902194876s" totalImagesPullingTime="8.897408672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:51.901715139 +0000 UTC m=+300.775986029" watchObservedRunningTime="2026-11-02 19:56:51.902194876 +0000 UTC m=+300.776465761"
Nov 02 19:56:52.350168 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.350083    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:52.354938 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.354915    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:52.642056 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.642017    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:52.642516 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.642082    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:52.642895 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.642873    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:56:52.643047 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:52.643029    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv"
Nov 02 19:56:56.658264 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:56.658233    5099 generic.go:356] "Generic (PLEG): container finished" podID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerID="8b554cd91b3473cea0c5286792e9f3ccb142465da4ddfc1c5568f243986ad6e2" exitCode=0
Nov 02 19:56:56.658636 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:56:56.658282    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerDied","Data":"8b554cd91b3473cea0c5286792e9f3ccb142465da4ddfc1c5568f243986ad6e2"}
Nov 02 19:57:04.695130 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:04.695089    5099 generic.go:356] "Generic (PLEG): container finished" podID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerID="10881dc47056f685eb51bd1763869eca5dbdd6cb576b76c04d67ce5880d3325e" exitCode=0
Nov 02 19:57:04.695535 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:04.695167    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerDied","Data":"10881dc47056f685eb51bd1763869eca5dbdd6cb576b76c04d67ce5880d3325e"}
Nov 02 19:57:05.843513 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.843486    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:57:05.938278 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.938242    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\" (UniqueName: \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\") pod \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") "
Nov 02 19:57:05.938449 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.938292    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\") pod \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") "
Nov 02 19:57:05.938449 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.938311    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\") pod \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\" (UID: \"5c393fe3-4bad-466d-aa7f-737c43e4bdd1\") "
Nov 02 19:57:05.939112 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.939083    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle" pod "5c393fe3-4bad-466d-aa7f-737c43e4bdd1" (UID: "5c393fe3-4bad-466d-aa7f-737c43e4bdd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:05.940497 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.940466    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr" pod "5c393fe3-4bad-466d-aa7f-737c43e4bdd1" (UID: "5c393fe3-4bad-466d-aa7f-737c43e4bdd1"). InnerVolumeSpecName "kube-api-access-nwkcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:57:05.943275 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:05.943254    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util" pod "5c393fe3-4bad-466d-aa7f-737c43e4bdd1" (UID: "5c393fe3-4bad-466d-aa7f-737c43e4bdd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:57:06.039484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.039390    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkcr\" (UniqueName: \"kubernetes.io/projected/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-kube-api-access-nwkcr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:06.039484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.039430    5099 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-util\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:06.039484 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.039440    5099 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c393fe3-4bad-466d-aa7f-737c43e4bdd1-bundle\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:57:06.705615 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.705583    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw"
Nov 02 19:57:06.705828 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.705610    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cqjvdw" event={"ID":"5c393fe3-4bad-466d-aa7f-737c43e4bdd1","Type":"ContainerDied","Data":"668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9"}
Nov 02 19:57:06.705828 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:06.705657    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="668e7916e8ddfaf439d6410c09c5dba911304e4b4e1f0a13dcc3373f2120d5d9"
Nov 02 19:57:16.092760 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:57:16.092719    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:57:33.784802 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.784744    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"]
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785094    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="util"
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785105    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="util"
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785116    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="extract"
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785121    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="extract"
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785133    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="pull"
Nov 02 19:57:33.785184 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785138    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="pull"
Nov 02 19:57:33.785398 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.785202    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c393fe3-4bad-466d-aa7f-737c43e4bdd1" containerName="extract"
Nov 02 19:57:33.791089 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.791068    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.843770 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.843739    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 19:57:33.843933 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.843917    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 19:57:33.874905 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.874870    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.875063 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.874946    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.875113 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.875072    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.875163 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.875128    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.875200 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.875167    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lvj\" (UniqueName: \"kubernetes.io/projected/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-kube-api-access-26lvj\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.875301 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.875283    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.893011 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.892984    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"]
Nov 02 19:57:33.938563 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.938534    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 19:57:33.938738 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.938615    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 19:57:33.938738 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.938689    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 19:57:33.938863 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.938748    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-km6nn\""
Nov 02 19:57:33.975994 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.975967    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976000    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lvj\" (UniqueName: \"kubernetes.io/projected/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-kube-api-access-26lvj\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976045    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976083    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976107    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976159 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976124    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.976582 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.976555    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.978689 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.978665    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.978817 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.978706    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.978867 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.978834    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:33.979016 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:33.978999    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:34.035640 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.035567    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 19:57:34.039295 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.039273    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lvj\" (UniqueName: \"kubernetes.io/projected/6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a-kube-api-access-26lvj\") pod \"opendatahub-operator-controller-manager-7846bf97f7-mrgrr\" (UID: \"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:34.066240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.066222    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.096224 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.096196    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 19:57:34.110162 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.110140    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:34.117449 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.117427    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 19:57:34.117548 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.117427    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 19:57:34.117548 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.117430    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7xhk4\""
Nov 02 19:57:34.177492 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.177451    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz5q\" (UniqueName: \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.177642 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.177579    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.278617 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.278580    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.278812 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.278746    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz5q\" (UniqueName: \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.279020 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.278996    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.342457 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.342392    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz5q\" (UniqueName: \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\") pod \"kuadrant-operator-controller-manager-678d87766f-rllqb\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.392099 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.392073    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:34.640964 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.640930    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"]
Nov 02 19:57:34.641551 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.641533    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 19:57:34.791355 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.791332    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 19:57:34.793523 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:57:34.793494    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c3ac23_06e8_4666_aa36_3b2a842f0305.slice/crio-a7d790a59cf5639eac168b53bb18fa7cbb9879bc7f30cb25307c8ef802c8d8c2 WatchSource:0}: Error finding container a7d790a59cf5639eac168b53bb18fa7cbb9879bc7f30cb25307c8ef802c8d8c2: Status 404 returned error can't find the container with id a7d790a59cf5639eac168b53bb18fa7cbb9879bc7f30cb25307c8ef802c8d8c2
Nov 02 19:57:34.818900 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.818873    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr" event={"ID":"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a","Type":"ContainerStarted","Data":"4f4792fa4f4ca59f7684b51da4614821b6b26cbe57d574ad34c1ae8ab0542300"}
Nov 02 19:57:34.820174 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:34.820150    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" event={"ID":"94c3ac23-06e8-4666-aa36-3b2a842f0305","Type":"ContainerStarted","Data":"a7d790a59cf5639eac168b53bb18fa7cbb9879bc7f30cb25307c8ef802c8d8c2"}
Nov 02 19:57:37.712295 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.712261    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"]
Nov 02 19:57:37.714648 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.714630    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:37.764008 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.763976    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 19:57:37.811010 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.810969    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvjk\" (UniqueName: \"kubernetes.io/projected/c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8-kube-api-access-dgvjk\") pod \"dns-operator-controller-manager-8557cbdb84-xmk4p\" (UID: \"c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:37.811270 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.811249    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9p2k8\""
Nov 02 19:57:37.813073 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.813049    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"]
Nov 02 19:57:37.911639 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.911601    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvjk\" (UniqueName: \"kubernetes.io/projected/c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8-kube-api-access-dgvjk\") pod \"dns-operator-controller-manager-8557cbdb84-xmk4p\" (UID: \"c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:37.966463 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:37.966388    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvjk\" (UniqueName: \"kubernetes.io/projected/c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8-kube-api-access-dgvjk\") pod \"dns-operator-controller-manager-8557cbdb84-xmk4p\" (UID: \"c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:38.029389 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:38.029350    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:38.398444 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:38.398413    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"]
Nov 02 19:57:38.402273 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:57:38.402245    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e2f5df_3ae1_4612_9c1a_ec5fc10479f8.slice/crio-96d62ed9b20d7123acb85773585e5fa02d28ddbf7cfbe10c6a4bd44aa3342162 WatchSource:0}: Error finding container 96d62ed9b20d7123acb85773585e5fa02d28ddbf7cfbe10c6a4bd44aa3342162: Status 404 returned error can't find the container with id 96d62ed9b20d7123acb85773585e5fa02d28ddbf7cfbe10c6a4bd44aa3342162
Nov 02 19:57:38.843774 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:38.843691    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p" event={"ID":"c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8","Type":"ContainerStarted","Data":"96d62ed9b20d7123acb85773585e5fa02d28ddbf7cfbe10c6a4bd44aa3342162"}
Nov 02 19:57:39.851083 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:39.851041    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr" event={"ID":"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a","Type":"ContainerStarted","Data":"c48e53ac3038146c64ace6d9cb2c450e0014225ca22b29e5fe9eeaf7424ad6b6"}
Nov 02 19:57:40.861610 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:40.859225    5099 generic.go:356] "Generic (PLEG): container finished" podID="6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a" containerID="c48e53ac3038146c64ace6d9cb2c450e0014225ca22b29e5fe9eeaf7424ad6b6" exitCode=0
Nov 02 19:57:40.861610 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:40.861133    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr" event={"ID":"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a","Type":"ContainerDied","Data":"c48e53ac3038146c64ace6d9cb2c450e0014225ca22b29e5fe9eeaf7424ad6b6"}
Nov 02 19:57:41.185635 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.185545    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"]
Nov 02 19:57:41.195654 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.195606    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:41.245461 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.245382    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"]
Nov 02 19:57:41.248120 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.248093    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-g2x26\""
Nov 02 19:57:41.347402 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.347362    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbzh\" (UniqueName: \"kubernetes.io/projected/bb3d21d6-d541-4a18-bf3f-339d2d830191-kube-api-access-4qbzh\") pod \"limitador-operator-controller-manager-5778bb4449-7gwdj\" (UID: \"bb3d21d6-d541-4a18-bf3f-339d2d830191\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:41.448400 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.448322    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbzh\" (UniqueName: \"kubernetes.io/projected/bb3d21d6-d541-4a18-bf3f-339d2d830191-kube-api-access-4qbzh\") pod \"limitador-operator-controller-manager-5778bb4449-7gwdj\" (UID: \"bb3d21d6-d541-4a18-bf3f-339d2d830191\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:41.503863 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.503828    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbzh\" (UniqueName: \"kubernetes.io/projected/bb3d21d6-d541-4a18-bf3f-339d2d830191-kube-api-access-4qbzh\") pod \"limitador-operator-controller-manager-5778bb4449-7gwdj\" (UID: \"bb3d21d6-d541-4a18-bf3f-339d2d830191\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:41.515295 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.515237    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:41.869223 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.869179    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr" event={"ID":"6cc1f319-f30e-4fa9-b5e7-ca2bb9a9585a","Type":"ContainerStarted","Data":"8d9acb2eb90046bfc55d8624fe0c2111ffcc76438952d06d12e2ec6cf72f8e70"}
Nov 02 19:57:41.869662 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.869526    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:41.909218 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.909180    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"]
Nov 02 19:57:41.912457 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:57:41.912424    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3d21d6_d541_4a18_bf3f_339d2d830191.slice/crio-d121b26d24fc276878a523d877ef57292c401b1c42cdca5d353749281d4acd6d WatchSource:0}: Error finding container d121b26d24fc276878a523d877ef57292c401b1c42cdca5d353749281d4acd6d: Status 404 returned error can't find the container with id d121b26d24fc276878a523d877ef57292c401b1c42cdca5d353749281d4acd6d
Nov 02 19:57:41.997664 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:41.997562    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr" podStartSLOduration=2.916886908 podStartE2EDuration="8.997544579s" totalImagesPullingTime="5.521980838s" totalInitContainerRuntime="558.676833ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:41.995375041 +0000 UTC m=+350.869645929" watchObservedRunningTime="2026-11-02 19:57:41.997544579 +0000 UTC m=+350.871815467"
Nov 02 19:57:42.877305 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:42.877262    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj" event={"ID":"bb3d21d6-d541-4a18-bf3f-339d2d830191","Type":"ContainerStarted","Data":"d121b26d24fc276878a523d877ef57292c401b1c42cdca5d353749281d4acd6d"}
Nov 02 19:57:42.879473 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:42.879445    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p" event={"ID":"c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8","Type":"ContainerStarted","Data":"68fb070a61f96b371abfb243247804524b86f6977b82acccb1ed42d288d38f59"}
Nov 02 19:57:42.879771 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:42.879748    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:42.993361 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:42.993299    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p" podStartSLOduration=1.907274994 podStartE2EDuration="5.993285162s" totalImagesPullingTime="4.086010168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:42.991819426 +0000 UTC m=+351.866090311" watchObservedRunningTime="2026-11-02 19:57:42.993285162 +0000 UTC m=+351.867556050"
Nov 02 19:57:43.885684 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:43.885646    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" event={"ID":"94c3ac23-06e8-4666-aa36-3b2a842f0305","Type":"ContainerStarted","Data":"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"}
Nov 02 19:57:43.886149 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:43.885891    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:43.999207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:43.999145    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" podStartSLOduration=2.217147368 podStartE2EDuration="10.999130847s" totalImagesPullingTime="8.781983479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:43.998030621 +0000 UTC m=+352.872301509" watchObservedRunningTime="2026-11-02 19:57:43.999130847 +0000 UTC m=+352.873401734"
Nov 02 19:57:46.097541 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:57:46.097501    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:57:46.906305 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:46.906276    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj" event={"ID":"bb3d21d6-d541-4a18-bf3f-339d2d830191","Type":"ContainerStarted","Data":"6b5dc897c9780cefa89cf782eae66629e16528de974ff13a3cc29a9c091943ff"}
Nov 02 19:57:46.906608 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:46.906345    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:57:47.014997 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:47.014933    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj" podStartSLOduration=1.503124841 podStartE2EDuration="6.014920114s" totalImagesPullingTime="4.511795273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:47.013820274 +0000 UTC m=+355.888091163" watchObservedRunningTime="2026-11-02 19:57:47.014920114 +0000 UTC m=+355.889191002"
Nov 02 19:57:52.881363 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:52.881330    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-mrgrr"
Nov 02 19:57:53.888621 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:53.888592    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xmk4p"
Nov 02 19:57:54.893834 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:54.893774    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 19:57:57.913044 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:57:57.913012    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gwdj"
Nov 02 19:58:07.611864 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.611825    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"]
Nov 02 19:58:07.615393 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.615372    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.666119 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.666094    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-58trp\""
Nov 02 19:58:07.674963 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.674937    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsnn\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-kube-api-access-pxsnn\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675118 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.674979    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675118 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675007    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675209 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675162    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675255 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675216    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675322 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675305    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675374 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675357    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675412 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675397    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.675455 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.675422    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.714885 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.714851    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"]
Nov 02 19:58:07.776563 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776526    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsnn\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-kube-api-access-pxsnn\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.776731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776580    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.776731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776619    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.776731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776653    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.776731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776690    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.776731 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776726    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776765    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.776822    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777103 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777088    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777247 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777112    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777247 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777161    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777318 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777285    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777358 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777313    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.777555 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.777533    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.779283 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.779254    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.779447 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.779431    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.831285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.831250    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.831427 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.831315    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsnn\" (UniqueName: \"kubernetes.io/projected/b3a55bd0-1b80-46f9-bfc4-fea42d0895a4-kube-api-access-pxsnn\") pod \"data-science-gateway-data-science-gateway-class-55fd485789hrswz\" (UID: \"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:07.927658 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:07.927587    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:08.338875 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:08.338851    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"]
Nov 02 19:58:08.340714 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:58:08.340687    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a55bd0_1b80_46f9_bfc4_fea42d0895a4.slice/crio-c42565e913e39f613750c340b6ad39b91b719f90ab9777a742311ebb6a9e526d WatchSource:0}: Error finding container c42565e913e39f613750c340b6ad39b91b719f90ab9777a742311ebb6a9e526d: Status 404 returned error can't find the container with id c42565e913e39f613750c340b6ad39b91b719f90ab9777a742311ebb6a9e526d
Nov 02 19:58:08.342933 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:08.342901    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:08.343006 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:08.342968    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:08.343006 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:08.342996    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:58:09.002651 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.002613    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz" event={"ID":"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4","Type":"ContainerStarted","Data":"f6fe6803937816b3c2faeabdbcb259f08ddc93f47ab120d2e003391785059e37"}
Nov 02 19:58:09.003064 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.002666    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz" event={"ID":"b3a55bd0-1b80-46f9-bfc4-fea42d0895a4","Type":"ContainerStarted","Data":"c42565e913e39f613750c340b6ad39b91b719f90ab9777a742311ebb6a9e526d"}
Nov 02 19:58:09.119281 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.119231    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz" podStartSLOduration=2.119216359 podStartE2EDuration="2.119216359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:09.119011719 +0000 UTC m=+377.993282608" watchObservedRunningTime="2026-11-02 19:58:09.119216359 +0000 UTC m=+377.993487246"
Nov 02 19:58:09.708811 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.708766    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:09.712380 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.712362    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.762817 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.762762    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8bwcw\""
Nov 02 19:58:09.762817 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.762762    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 19:58:09.794880 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.794837    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.794880 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.794878    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqqs\" (UniqueName: \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.811290 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.811261    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:09.895589 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.895559    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.895761 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.895594    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqqs\" (UniqueName: \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.896240 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.896218    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:09.928028 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.927998    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:09.932645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.932623    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:09.944261 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.944228    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:09.952472 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:09.952442    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqqs\" (UniqueName: \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\") pod \"limitador-limitador-74dfcfb894-knr5q\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:10.007155 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:10.007077    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:10.008042 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:10.008019    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-55fd485789hrswz"
Nov 02 19:58:10.026214 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:10.026190    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:10.392960 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:10.392936    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:10.394980 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:58:10.394946    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67367e9d_c636_4de0_b40d_76f306f35912.slice/crio-2553c2d0b3cd5ec1984b5fbf6cb96b76a1f18dacd109dcf4393c2dc8e7f2b312 WatchSource:0}: Error finding container 2553c2d0b3cd5ec1984b5fbf6cb96b76a1f18dacd109dcf4393c2dc8e7f2b312: Status 404 returned error can't find the container with id 2553c2d0b3cd5ec1984b5fbf6cb96b76a1f18dacd109dcf4393c2dc8e7f2b312
Nov 02 19:58:11.012038 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:11.012003    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" event={"ID":"67367e9d-c636-4de0-b40d-76f306f35912","Type":"ContainerStarted","Data":"2553c2d0b3cd5ec1984b5fbf6cb96b76a1f18dacd109dcf4393c2dc8e7f2b312"}
Nov 02 19:58:12.776660 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.776628    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:58:12.781344 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.781322    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:58:12.818848 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.818818    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhd9\" (UniqueName: \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\") pod \"authorino-56c6c9ddd5-wp4z2\" (UID: \"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3\") " pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:58:12.830383 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.830358    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:58:12.876071 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.876043    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5v79g\""
Nov 02 19:58:12.919319 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.919291    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhd9\" (UniqueName: \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\") pod \"authorino-56c6c9ddd5-wp4z2\" (UID: \"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3\") " pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:58:12.974877 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:12.974844    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhd9\" (UniqueName: \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\") pod \"authorino-56c6c9ddd5-wp4z2\" (UID: \"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3\") " pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:58:13.106432 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:13.106360    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:58:13.474533 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:13.474511    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:58:13.476769 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:58:13.476744    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bd681e5_ed9b_4fe1_bb12_07840b2a1ad3.slice/crio-07084dd401f50ba83cc6d827dcf60fa01cd5fc75790fc508a2454aeca9624694 WatchSource:0}: Error finding container 07084dd401f50ba83cc6d827dcf60fa01cd5fc75790fc508a2454aeca9624694: Status 404 returned error can't find the container with id 07084dd401f50ba83cc6d827dcf60fa01cd5fc75790fc508a2454aeca9624694
Nov 02 19:58:14.027018 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:14.026982    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" event={"ID":"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3","Type":"ContainerStarted","Data":"07084dd401f50ba83cc6d827dcf60fa01cd5fc75790fc508a2454aeca9624694"}
Nov 02 19:58:15.033492 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:15.033447    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" event={"ID":"67367e9d-c636-4de0-b40d-76f306f35912","Type":"ContainerStarted","Data":"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"}
Nov 02 19:58:15.033983 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:15.033626    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:15.142021 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:15.141971    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" podStartSLOduration=1.9809255540000001 podStartE2EDuration="6.141954538s" totalImagesPullingTime="4.161028984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:15.140953862 +0000 UTC m=+384.015224750" watchObservedRunningTime="2026-11-02 19:58:15.141954538 +0000 UTC m=+384.016225459"
Nov 02 19:58:16.096841 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:58:16.096807    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:58:17.044360 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:17.044321    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" event={"ID":"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3","Type":"ContainerStarted","Data":"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"}
Nov 02 19:58:17.150907 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:17.150850    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" podStartSLOduration=2.023831315 podStartE2EDuration="5.150835725s" totalImagesPullingTime="3.12700441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:17.149366873 +0000 UTC m=+386.023637762" watchObservedRunningTime="2026-11-02 19:58:17.150835725 +0000 UTC m=+386.025106610"
Nov 02 19:58:18.146076 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.146043    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"]
Nov 02 19:58:18.149700 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.149670    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.199833 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.199807    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-fdcbh\""
Nov 02 19:58:18.200164 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.199872    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 19:58:18.200164 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.199922    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 19:58:18.268602 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.268563    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.268837 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.268609    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ht9\" (UniqueName: \"kubernetes.io/projected/3bd3085e-d212-4c20-bdce-605011614404-kube-api-access-x4ht9\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.268837 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.268683    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bd3085e-d212-4c20-bdce-605011614404-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.268837 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.268728    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.343012 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.342980    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 19:58:18.344645 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.344621    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"]
Nov 02 19:58:18.369207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369166    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.369400 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369228    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ht9\" (UniqueName: \"kubernetes.io/projected/3bd3085e-d212-4c20-bdce-605011614404-kube-api-access-x4ht9\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.369400 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369265    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bd3085e-d212-4c20-bdce-605011614404-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.369400 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369319    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.369658 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369631    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.369711 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.369647    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3bd3085e-d212-4c20-bdce-605011614404-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.370209 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.370189    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bd3085e-d212-4c20-bdce-605011614404-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.423131 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.423072    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ht9\" (UniqueName: \"kubernetes.io/projected/3bd3085e-d212-4c20-bdce-605011614404-kube-api-access-x4ht9\") pod \"kserve-module-controller-manager-7566cf5fc8-kbzxz\" (UID: \"3bd3085e-d212-4c20-bdce-605011614404\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.470170 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.470145    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:18.879207 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:18.879179    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"]
Nov 02 19:58:18.880467 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:58:18.880423    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd3085e_d212_4c20_bdce_605011614404.slice/crio-662dba1cb05cbb07f60b8b52ff8ad1ae72e3a2675740dd4921589d5e9df76191 WatchSource:0}: Error finding container 662dba1cb05cbb07f60b8b52ff8ad1ae72e3a2675740dd4921589d5e9df76191: Status 404 returned error can't find the container with id 662dba1cb05cbb07f60b8b52ff8ad1ae72e3a2675740dd4921589d5e9df76191
Nov 02 19:58:19.055526 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:19.055489    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz" event={"ID":"3bd3085e-d212-4c20-bdce-605011614404","Type":"ContainerStarted","Data":"662dba1cb05cbb07f60b8b52ff8ad1ae72e3a2675740dd4921589d5e9df76191"}
Nov 02 19:58:24.082695 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:24.082656    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz" event={"ID":"3bd3085e-d212-4c20-bdce-605011614404","Type":"ContainerStarted","Data":"f906d277cecf10f92a397ca0afe82cacb51d6293e1c78ab4b1e82e77b47486e0"}
Nov 02 19:58:24.083148 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:24.082745    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:24.240137 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:24.240088    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz" podStartSLOduration=1.631150119 podStartE2EDuration="6.24007389s" totalImagesPullingTime="4.608923771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:24.238636184 +0000 UTC m=+393.112907072" watchObservedRunningTime="2026-11-02 19:58:24.24007389 +0000 UTC m=+393.114344823"
Nov 02 19:58:26.040024 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:26.039994    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:27.130241 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.129223    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:27.130241 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.129536    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" podUID="67367e9d-c636-4de0-b40d-76f306f35912" containerName="limitador" containerID="cri-o://9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482" gracePeriod=30
Nov 02 19:58:27.682517 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.682494    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:27.751256 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.751174    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\" (UniqueName: \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\") pod \"67367e9d-c636-4de0-b40d-76f306f35912\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") "
Nov 02 19:58:27.751416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.751289    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\" (UniqueName: \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\") pod \"67367e9d-c636-4de0-b40d-76f306f35912\" (UID: \"67367e9d-c636-4de0-b40d-76f306f35912\") "
Nov 02 19:58:27.751611 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.751590    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file" pod "67367e9d-c636-4de0-b40d-76f306f35912" (UID: "67367e9d-c636-4de0-b40d-76f306f35912"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:58:27.753349 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.753328    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs" pod "67367e9d-c636-4de0-b40d-76f306f35912" (UID: "67367e9d-c636-4de0-b40d-76f306f35912"). InnerVolumeSpecName "kube-api-access-dhqqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:58:27.852533 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.852501    5099 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67367e9d-c636-4de0-b40d-76f306f35912-config-file\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:58:27.852697 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:27.852551    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqqs\" (UniqueName: \"kubernetes.io/projected/67367e9d-c636-4de0-b40d-76f306f35912-kube-api-access-dhqqs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:58:28.103435 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.103356    5099 generic.go:356] "Generic (PLEG): container finished" podID="67367e9d-c636-4de0-b40d-76f306f35912" containerID="9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482" exitCode=0
Nov 02 19:58:28.103435 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.103400    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" event={"ID":"67367e9d-c636-4de0-b40d-76f306f35912","Type":"ContainerDied","Data":"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"}
Nov 02 19:58:28.103435 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.103409    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q"
Nov 02 19:58:28.103634 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.103451    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-knr5q" event={"ID":"67367e9d-c636-4de0-b40d-76f306f35912","Type":"ContainerDied","Data":"2553c2d0b3cd5ec1984b5fbf6cb96b76a1f18dacd109dcf4393c2dc8e7f2b312"}
Nov 02 19:58:28.103634 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.103497    5099 scope.go:122] "RemoveContainer" containerID="9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"
Nov 02 19:58:28.112662 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.112643    5099 scope.go:122] "RemoveContainer" containerID="9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"
Nov 02 19:58:28.112930 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:58:28.112912    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482\": container with ID starting with 9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482 not found: ID does not exist" containerID="9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"
Nov 02 19:58:28.112992 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.112940    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482"} err="failed to get container status \"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482\": rpc error: code = NotFound desc = could not find container \"9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482\": container with ID starting with 9366ad9fd268177a405d603a2b433d523c9ddcfd006f6f42db06623b45d35482 not found: ID does not exist"
Nov 02 19:58:28.257322 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.257290    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:28.304428 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:28.304387    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-knr5q"]
Nov 02 19:58:29.255483 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:29.255438    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67367e9d-c636-4de0-b40d-76f306f35912" path="/var/lib/kubelet/pods/67367e9d-c636-4de0-b40d-76f306f35912/volumes"
Nov 02 19:58:32.899138 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.899060    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"]
Nov 02 19:58:32.899491 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.899465    5099 cpu_manager.go:414] "containerMap: removing container" podUID="67367e9d-c636-4de0-b40d-76f306f35912" containerName="limitador"
Nov 02 19:58:32.899491 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.899476    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67367e9d-c636-4de0-b40d-76f306f35912" containerName="limitador"
Nov 02 19:58:32.899568 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.899544    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="67367e9d-c636-4de0-b40d-76f306f35912" containerName="limitador"
Nov 02 19:58:32.902600 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.902579    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:32.952724 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.952674    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:58:32.954678 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.954654    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"]
Nov 02 19:58:32.998748 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.998718    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14e3d047-8c95-4241-b899-20a8466fd914-data\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:32.998904 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.998761    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzj4\" (UniqueName: \"kubernetes.io/projected/14e3d047-8c95-4241-b899-20a8466fd914-kube-api-access-fqzj4\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:32.999870 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.999849    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:58:32.999934 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.999852    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 19:58:32.999980 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:32.999939    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ksgm9\""
Nov 02 19:58:33.099945 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.099907    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14e3d047-8c95-4241-b899-20a8466fd914-data\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:33.100117 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.099955    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzj4\" (UniqueName: \"kubernetes.io/projected/14e3d047-8c95-4241-b899-20a8466fd914-kube-api-access-fqzj4\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:33.100323 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.100304    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/14e3d047-8c95-4241-b899-20a8466fd914-data\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:33.153871 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.153769    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzj4\" (UniqueName: \"kubernetes.io/projected/14e3d047-8c95-4241-b899-20a8466fd914-kube-api-access-fqzj4\") pod \"postgres-7cdbf6dc44-qvnk8\" (UID: \"14e3d047-8c95-4241-b899-20a8466fd914\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:33.214877 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.214843    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:33.575956 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:33.575932    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"]
Nov 02 19:58:33.577706 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:58:33.577676    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e3d047_8c95_4241_b899_20a8466fd914.slice/crio-c4747126c1e0c3635d9b769e653c61de1ccba10adddd22ba1b07bd87deae5bd8 WatchSource:0}: Error finding container c4747126c1e0c3635d9b769e653c61de1ccba10adddd22ba1b07bd87deae5bd8: Status 404 returned error can't find the container with id c4747126c1e0c3635d9b769e653c61de1ccba10adddd22ba1b07bd87deae5bd8
Nov 02 19:58:34.133412 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:34.133370    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8" event={"ID":"14e3d047-8c95-4241-b899-20a8466fd914","Type":"ContainerStarted","Data":"c4747126c1e0c3635d9b769e653c61de1ccba10adddd22ba1b07bd87deae5bd8"}
Nov 02 19:58:41.170588 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:41.170552    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8" event={"ID":"14e3d047-8c95-4241-b899-20a8466fd914","Type":"ContainerStarted","Data":"09b833b4f0e82f997321bccf5fbc877d99cf8019b85ee56a49a3098a21cab9bf"}
Nov 02 19:58:41.171056 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:41.170619    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:58:41.277932 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:41.277884    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8" podStartSLOduration=2.378943668 podStartE2EDuration="9.27787051s" totalImagesPullingTime="6.898926842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:41.277051889 +0000 UTC m=+410.151322780" watchObservedRunningTime="2026-11-02 19:58:41.27787051 +0000 UTC m=+410.152141397"
Nov 02 19:58:45.088518 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:45.088485    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kbzxz"
Nov 02 19:58:46.092363 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:58:46.092317    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:58:47.204511 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:58:47.204486    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qvnk8"
Nov 02 19:59:16.092999 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:59:16.092964    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:59:19.363138 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.363106    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"]
Nov 02 19:59:19.388944 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.388917    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.439285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.439260    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 19:59:19.439285 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.439265    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 19:59:19.439503 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.439289    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 19:59:19.439503 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.439336    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-42zng\""
Nov 02 19:59:19.488621 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.488589    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"]
Nov 02 19:59:19.515747 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515714    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.515924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515759    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ph5\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-kube-api-access-r9ph5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.515924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515816    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.515924 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515902    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.516029 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515924    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ddef60-e0ce-4a6a-924b-3224709e79d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.516029 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515950    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.516029 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.515975    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617161 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617085    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617161 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617122    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ddef60-e0ce-4a6a-924b-3224709e79d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617161 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617151    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617381 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617191    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617381 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617343    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617488 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617406    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ph5\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-kube-api-access-r9ph5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.617488 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.617441    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.618131 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.618107    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.619409 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.619389    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ddef60-e0ce-4a6a-924b-3224709e79d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.619558 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.619538    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.619915 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.619897    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.619995 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.619921    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.671308 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.671280    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ph5\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-kube-api-access-r9ph5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.671650 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.671630    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ddef60-e0ce-4a6a-924b-3224709e79d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j8j94\" (UID: \"70ddef60-e0ce-4a6a-924b-3224709e79d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:19.701728 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:19.701702    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:20.115075 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:20.115052    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"]
Nov 02 19:59:20.117193 ip-10-0-133-26 kubenswrapper[5099]: W1102 19:59:20.117155    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ddef60_e0ce_4a6a_924b_3224709e79d6.slice/crio-67d89b1ca8615088e4b4bee423a12037d9e57fe4228838e68609c48a53535c05 WatchSource:0}: Error finding container 67d89b1ca8615088e4b4bee423a12037d9e57fe4228838e68609c48a53535c05: Status 404 returned error can't find the container with id 67d89b1ca8615088e4b4bee423a12037d9e57fe4228838e68609c48a53535c05
Nov 02 19:59:20.358221 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:20.358191    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94" event={"ID":"70ddef60-e0ce-4a6a-924b-3224709e79d6","Type":"ContainerStarted","Data":"67d89b1ca8615088e4b4bee423a12037d9e57fe4228838e68609c48a53535c05"}
Nov 02 19:59:26.125754 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:26.125720    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:26.126060 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:26.125815    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:26.389647 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:26.389561    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94" event={"ID":"70ddef60-e0ce-4a6a-924b-3224709e79d6","Type":"ContainerStarted","Data":"d7427bc10d2bf7bd7fa838789f18a5fe01fa94d7886861c9e6ade6d627b94cb4"}
Nov 02 19:59:26.389808 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:26.389689    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:26.548272 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:26.548215    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94" podStartSLOduration=1.542594641 podStartE2EDuration="7.548200856s" totalImagesPullingTime="6.005606215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:26.500376314 +0000 UTC m=+455.374647203" watchObservedRunningTime="2026-11-02 19:59:26.548200856 +0000 UTC m=+455.422471743"
Nov 02 19:59:27.395775 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:27.395745    5099 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j8j94 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 19:59:27.396172 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:27.395805    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94" podUID="70ddef60-e0ce-4a6a-924b-3224709e79d6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 19:59:30.395445 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:30.395406    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j8j94"
Nov 02 19:59:45.501384 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.501351    5099 generic.go:356] "Generic (PLEG): container finished" podID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerID="02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036" exitCode=0
Nov 02 19:59:45.501864 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.501411    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerDied","Data":"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"}
Nov 02 19:59:45.501864 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.501761    5099 scope.go:122] "RemoveContainer" containerID="02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"
Nov 02 19:59:45.503061 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.503024    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:45.503161 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.503120    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:45.503202 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:45.503171    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 19:59:46.093164 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:59:46.093133    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 19:59:46.507904 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:46.507870    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerStarted","Data":"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"}
Nov 02 19:59:47.358299 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:47.358217    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:59:47.358609 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:47.358583    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jbvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:59:47.358822 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:47.358768    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 02 19:59:48.351224 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:48.351188    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jbvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:59:48.351860 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:48.351824    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 02 19:59:49.121195 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.121156    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:59:49.121369 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.121351    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" podUID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" containerName="authorino" containerID="cri-o://60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff" gracePeriod=30
Nov 02 19:59:49.379118 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.378529    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jbvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:59:49.379118 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.378600    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 02 19:59:49.435753 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.435729    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:59:49.487416 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.487382    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\" (UniqueName: \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\") pod \"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3\" (UID: \"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3\") "
Nov 02 19:59:49.489404 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.489378    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9" pod "8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" (UID: "8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3"). InnerVolumeSpecName "kube-api-access-bwhd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:59:49.522734 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.522704    5099 generic.go:356] "Generic (PLEG): container finished" podID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" containerID="60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff" exitCode=0
Nov 02 19:59:49.522912 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.522754    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2"
Nov 02 19:59:49.522912 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.522810    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" event={"ID":"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3","Type":"ContainerDied","Data":"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"}
Nov 02 19:59:49.522912 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.522855    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wp4z2" event={"ID":"8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3","Type":"ContainerDied","Data":"07084dd401f50ba83cc6d827dcf60fa01cd5fc75790fc508a2454aeca9624694"}
Nov 02 19:59:49.522912 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.522876    5099 scope.go:122] "RemoveContainer" containerID="60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"
Nov 02 19:59:49.531625 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.531611    5099 scope.go:122] "RemoveContainer" containerID="60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"
Nov 02 19:59:49.531920 ip-10-0-133-26 kubenswrapper[5099]: E1102 19:59:49.531901    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff\": container with ID starting with 60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff not found: ID does not exist" containerID="60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"
Nov 02 19:59:49.531975 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.531930    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff"} err="failed to get container status \"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff\": rpc error: code = NotFound desc = could not find container \"60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff\": container with ID starting with 60f868113942c2392c99da22d29c0c2c2d6295bd21937db7c3d01b86aa8c78ff not found: ID does not exist"
Nov 02 19:59:49.588844 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.588809    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwhd9\" (UniqueName: \"kubernetes.io/projected/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3-kube-api-access-bwhd9\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 19:59:49.663922 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.663827    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:59:49.667039 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:49.667012    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wp4z2"]
Nov 02 19:59:50.358201 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:50.358163    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jbvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:59:50.358477 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:50.358446    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 02 19:59:51.266512 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:51.266482    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" path="/var/lib/kubelet/pods/8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3/volumes"
Nov 02 19:59:51.362005 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:51.361976    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jbvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 02 19:59:51.362170 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:51.362029    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 02 19:59:52.354080 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:52.354053    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:59:52.354501 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:52.354413    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 19:59:52.355064 ip-10-0-133-26 kubenswrapper[5099]: I1102 19:59:52.355041    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 20:00:16.093207 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:00:16.093122    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:00:46.093250 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:00:46.093215    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:01:16.092879 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:01:16.092843    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:01:46.091214 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:01:46.091132    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:01:51.317556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:01:51.317500    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:01:51.319046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:01:51.319026    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:02:01.017906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.017871    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"]
Nov 02 20:02:01.018448 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.018427    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" containerName="authorino"
Nov 02 20:02:01.018534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.018450    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" containerName="authorino"
Nov 02 20:02:01.018588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.018561    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd681e5-ed9b-4fe1-bb12-07840b2a1ad3" containerName="authorino"
Nov 02 20:02:01.021869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.021846    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.119315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.119275    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d2qwk\""
Nov 02 20:02:01.119315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.119278    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 20:02:01.119315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.119282    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 20:02:01.119537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.119326    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 20:02:01.121326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.121304    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"]
Nov 02 20:02:01.123578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123555    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2bh\" (UniqueName: \"kubernetes.io/projected/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-kube-api-access-6b2bh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.123676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123615    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.123753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123731    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.123854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123826    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.123970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123883    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.123970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.123909    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225247    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225285    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225325    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2bh\" (UniqueName: \"kubernetes.io/projected/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-kube-api-access-6b2bh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225357    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225377    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225437    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.225763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.225714    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.226078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.226034    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.226331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.226307    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.228302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.228271    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.244656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.244615    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.284329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.284257    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2bh\" (UniqueName: \"kubernetes.io/projected/8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7-kube-api-access-6b2bh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7bnfm\" (UID: \"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.334381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.334350    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:01.702208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:01.702177    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"]
Nov 02 20:02:01.704690 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:02:01.704655    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4c9172_9ae9_48bd_a3b5_ecd59c5217c7.slice/crio-8bf118cd3bdbb86255cc80cb378059a98c0941bfc8fa22c0b64ca828e17b1ed4 WatchSource:0}: Error finding container 8bf118cd3bdbb86255cc80cb378059a98c0941bfc8fa22c0b64ca828e17b1ed4: Status 404 returned error can't find the container with id 8bf118cd3bdbb86255cc80cb378059a98c0941bfc8fa22c0b64ca828e17b1ed4
Nov 02 20:02:02.138179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:02.138140    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm" event={"ID":"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7","Type":"ContainerStarted","Data":"8bf118cd3bdbb86255cc80cb378059a98c0941bfc8fa22c0b64ca828e17b1ed4"}
Nov 02 20:02:04.153531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:04.153494    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm" event={"ID":"8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7","Type":"ContainerStarted","Data":"4fd81e4fa63c867abc1b52cdc65ba87b25813393a86e8abe8f9ec5db72d16729"}
Nov 02 20:02:04.262377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:04.262325    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm" podStartSLOduration=2.722221373 podStartE2EDuration="4.262312662s" totalImagesPullingTime="1.540091289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:04.261593334 +0000 UTC m=+613.135864222" watchObservedRunningTime="2026-11-02 20:02:04.262312662 +0000 UTC m=+613.136583550"
Nov 02 20:02:04.335178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:04.335145    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:07.917907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.917869    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"]
Nov 02 20:02:07.921491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.921470    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.984813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.983897    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"]
Nov 02 20:02:07.992010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.991972    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.992167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.992036    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.992167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.992092    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hl4\" (UniqueName: \"kubernetes.io/projected/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-kube-api-access-27hl4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.992167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.992139    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.992394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.992223    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:07.992394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:07.992260    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.027484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.027449    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 20:02:08.095629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095597    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.095881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095665    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.095881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095695    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.095881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095744    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.095881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095776    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.095881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.095853    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hl4\" (UniqueName: \"kubernetes.io/projected/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-kube-api-access-27hl4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.096209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.096170    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.096209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.096175    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.096462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.096436    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.098615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.098589    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.098735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.098624    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.150206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.150160    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hl4\" (UniqueName: \"kubernetes.io/projected/8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26-kube-api-access-27hl4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6\" (UID: \"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.233830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.233716    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:08.606927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:08.606899    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"]
Nov 02 20:02:08.608812 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:02:08.608749    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8954ba67_1f7e_4ab0_b3cf_3c88b9da5f26.slice/crio-b55f0d5372fc859ffad1b7c796c6cca35b67b8207202a3ff55f047399acfda91 WatchSource:0}: Error finding container b55f0d5372fc859ffad1b7c796c6cca35b67b8207202a3ff55f047399acfda91: Status 404 returned error can't find the container with id b55f0d5372fc859ffad1b7c796c6cca35b67b8207202a3ff55f047399acfda91
Nov 02 20:02:09.179083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:09.179043    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6" event={"ID":"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26","Type":"ContainerStarted","Data":"b55f0d5372fc859ffad1b7c796c6cca35b67b8207202a3ff55f047399acfda91"}
Nov 02 20:02:10.184969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:10.184931    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6" event={"ID":"8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26","Type":"ContainerStarted","Data":"61472cd9d112f5134ef74419cb58b3f2d5621c3d307f32a025cfb46ce907169e"}
Nov 02 20:02:10.234313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:10.234276    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:10.294122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:10.294071    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6" podStartSLOduration=2.6618508849999998 podStartE2EDuration="3.294055618s" totalImagesPullingTime="632.204733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:10.292423257 +0000 UTC m=+619.166694155" watchObservedRunningTime="2026-11-02 20:02:10.294055618 +0000 UTC m=+619.168326506"
Nov 02 20:02:11.335025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:11.334993    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:11.347502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:11.347475    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:12.206491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:12.206463    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7bnfm"
Nov 02 20:02:16.093904 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:02:16.093868    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:02:18.234197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:18.234169    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:18.246904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:18.246863    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:19.242764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:19.242732    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6"
Nov 02 20:02:45.258209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.258179    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:02:45.260704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.260680    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.309409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.309373    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:02:45.327282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.327237    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.327427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.327289    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95h4\" (UniqueName: \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.354222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.354196    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 20:02:45.354380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.354240    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mfch7\""
Nov 02 20:02:45.354380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.354245    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 20:02:45.428698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.428667    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.428872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.428704    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95h4\" (UniqueName: \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.431196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.431175    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.482873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.482842    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95h4\" (UniqueName: \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\") pod \"ai-gateway-controller-7db955b5ff-8sr8j\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.574206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.574130    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:45.940185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.940162    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:02:45.942114 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:02:45.942091    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69e49d4_98eb_4b62_ae8a_80bfb53f2831.slice/crio-d2ab8d64adb69f25d86f1b074e3947eefb8708df470425df067e7e84b6bbed6d WatchSource:0}: Error finding container d2ab8d64adb69f25d86f1b074e3947eefb8708df470425df067e7e84b6bbed6d: Status 404 returned error can't find the container with id d2ab8d64adb69f25d86f1b074e3947eefb8708df470425df067e7e84b6bbed6d
Nov 02 20:02:45.943508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:45.943488    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:02:46.092472 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:02:46.092436    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:02:46.363063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:46.363030    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" event={"ID":"f69e49d4-98eb-4b62-ae8a-80bfb53f2831","Type":"ContainerStarted","Data":"d2ab8d64adb69f25d86f1b074e3947eefb8708df470425df067e7e84b6bbed6d"}
Nov 02 20:02:52.509506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:52.509478    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 20:02:53.397434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:53.397396    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" event={"ID":"f69e49d4-98eb-4b62-ae8a-80bfb53f2831","Type":"ContainerStarted","Data":"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"}
Nov 02 20:02:53.397601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:53.397480    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:02:53.504976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:53.504920    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" podStartSLOduration=1.988249396 podStartE2EDuration="8.504907496s" totalImagesPullingTime="6.5166581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:53.504324685 +0000 UTC m=+662.378595573" watchObservedRunningTime="2026-11-02 20:02:53.504907496 +0000 UTC m=+662.379178387"
Nov 02 20:02:56.756616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.756577    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"]
Nov 02 20:02:56.760243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.760224    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.825654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.825618    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"]
Nov 02 20:02:56.838656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.838621    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ce931f-cff8-4e59-8c3e-387889e247c0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.838877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.838774    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pn8f\" (UniqueName: \"kubernetes.io/projected/83ce931f-cff8-4e59-8c3e-387889e247c0-kube-api-access-9pn8f\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.869457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.869428    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 20:02:56.869457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.869434    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-stgwf\""
Nov 02 20:02:56.922565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.922532    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"]
Nov 02 20:02:56.926142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.926123    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:56.939854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.939833    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ce931f-cff8-4e59-8c3e-387889e247c0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.939960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.939900    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pn8f\" (UniqueName: \"kubernetes.io/projected/83ce931f-cff8-4e59-8c3e-387889e247c0-kube-api-access-9pn8f\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.940495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.940476    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ce931f-cff8-4e59-8c3e-387889e247c0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:56.977430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.977406    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"]
Nov 02 20:02:56.993980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:56.993949    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pn8f\" (UniqueName: \"kubernetes.io/projected/83ce931f-cff8-4e59-8c3e-387889e247c0-kube-api-access-9pn8f\") pod \"payload-pre-processing-7b98bb678c-tm9g5\" (UID: \"83ce931f-cff8-4e59-8c3e-387889e247c0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:57.040960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.040876    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvnwb\" (UniqueName: \"kubernetes.io/projected/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-kube-api-access-dvnwb\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.040960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.040925    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.072834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.072807    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:02:57.142392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.142359    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvnwb\" (UniqueName: \"kubernetes.io/projected/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-kube-api-access-dvnwb\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.142392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.142397    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.143157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.143135    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.195845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.195817    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvnwb\" (UniqueName: \"kubernetes.io/projected/33c764f6-f4d1-4c8c-89b0-e7df0291baa9-kube-api-access-dvnwb\") pod \"payload-processing-55fbbbf9dc-5fzvg\" (UID: \"33c764f6-f4d1-4c8c-89b0-e7df0291baa9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.239258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.239225    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:02:57.433765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.433735    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"]
Nov 02 20:02:57.435635 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:02:57.435602    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ce931f_cff8_4e59_8c3e_387889e247c0.slice/crio-00f304b7e04a8be0c233ceb60537fb97e6f6c27036433f9243bdc550cbe42056 WatchSource:0}: Error finding container 00f304b7e04a8be0c233ceb60537fb97e6f6c27036433f9243bdc550cbe42056: Status 404 returned error can't find the container with id 00f304b7e04a8be0c233ceb60537fb97e6f6c27036433f9243bdc550cbe42056
Nov 02 20:02:57.605119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:57.605094    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"]
Nov 02 20:02:57.607197 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:02:57.607171    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c764f6_f4d1_4c8c_89b0_e7df0291baa9.slice/crio-4c5f9ffc174ee0ce50a9012d426a3dbea1ba945ed3d8b7a120b1c8f025044221 WatchSource:0}: Error finding container 4c5f9ffc174ee0ce50a9012d426a3dbea1ba945ed3d8b7a120b1c8f025044221: Status 404 returned error can't find the container with id 4c5f9ffc174ee0ce50a9012d426a3dbea1ba945ed3d8b7a120b1c8f025044221
Nov 02 20:02:58.423396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:58.423362    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5" event={"ID":"83ce931f-cff8-4e59-8c3e-387889e247c0","Type":"ContainerStarted","Data":"00f304b7e04a8be0c233ceb60537fb97e6f6c27036433f9243bdc550cbe42056"}
Nov 02 20:02:58.424605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:02:58.424582    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg" event={"ID":"33c764f6-f4d1-4c8c-89b0-e7df0291baa9","Type":"ContainerStarted","Data":"4c5f9ffc174ee0ce50a9012d426a3dbea1ba945ed3d8b7a120b1c8f025044221"}
Nov 02 20:03:01.442793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.442750    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5" event={"ID":"83ce931f-cff8-4e59-8c3e-387889e247c0","Type":"ContainerStarted","Data":"65d923dd9d5351a4783470e365da7ba8bca7626e17bdc97ef67ff5476a94419b"}
Nov 02 20:03:01.443261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.442827    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:03:01.444440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.444419    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg" event={"ID":"33c764f6-f4d1-4c8c-89b0-e7df0291baa9","Type":"ContainerStarted","Data":"4977e4a31b5475ac48177fd22f6701e9a1af96a6f660162ad7cee619bc2bcd72"}
Nov 02 20:03:01.444553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.444544    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:03:01.550224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.550174    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5" podStartSLOduration=2.4039207129999998 podStartE2EDuration="5.550161642s" totalImagesPullingTime="3.146240929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:01.549578193 +0000 UTC m=+670.423849134" watchObservedRunningTime="2026-11-02 20:03:01.550161642 +0000 UTC m=+670.424432530"
Nov 02 20:03:01.659643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:01.659580    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg" podStartSLOduration=2.582414919 podStartE2EDuration="5.659557337s" totalImagesPullingTime="3.077142418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:01.656736714 +0000 UTC m=+670.531007604" watchObservedRunningTime="2026-11-02 20:03:01.659557337 +0000 UTC m=+670.533828226"
Nov 02 20:03:04.404550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.404522    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:03:04.727484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.727405    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-696cfc768f-hhwjb"]
Nov 02 20:03:04.731048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.731026    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.783222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.783190    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-696cfc768f-hhwjb"]
Nov 02 20:03:04.818646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.818617    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgtt\" (UniqueName: \"kubernetes.io/projected/e3843dff-d849-4187-a2ae-a65de3254375-kube-api-access-nfgtt\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.818821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.818654    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3843dff-d849-4187-a2ae-a65de3254375-metrics-certs\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.920234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.920200    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgtt\" (UniqueName: \"kubernetes.io/projected/e3843dff-d849-4187-a2ae-a65de3254375-kube-api-access-nfgtt\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.920392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.920239    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3843dff-d849-4187-a2ae-a65de3254375-metrics-certs\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.922602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.922574    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3843dff-d849-4187-a2ae-a65de3254375-metrics-certs\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:04.973646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:04.973618    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgtt\" (UniqueName: \"kubernetes.io/projected/e3843dff-d849-4187-a2ae-a65de3254375-kube-api-access-nfgtt\") pod \"ai-gateway-controller-696cfc768f-hhwjb\" (UID: \"e3843dff-d849-4187-a2ae-a65de3254375\") " pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:05.043642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:05.043564    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:05.404948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:05.404924    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-696cfc768f-hhwjb"]
Nov 02 20:03:05.406766 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:05.406733    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3843dff_d849_4187_a2ae_a65de3254375.slice/crio-ea59cd19a72d3f3a47435f33e103980f89f929013fe39bf06fe8bb3f6f5e6be5 WatchSource:0}: Error finding container ea59cd19a72d3f3a47435f33e103980f89f929013fe39bf06fe8bb3f6f5e6be5: Status 404 returned error can't find the container with id ea59cd19a72d3f3a47435f33e103980f89f929013fe39bf06fe8bb3f6f5e6be5
Nov 02 20:03:05.463410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:05.463380    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb" event={"ID":"e3843dff-d849-4187-a2ae-a65de3254375","Type":"ContainerStarted","Data":"ea59cd19a72d3f3a47435f33e103980f89f929013fe39bf06fe8bb3f6f5e6be5"}
Nov 02 20:03:06.475767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:06.475730    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb" event={"ID":"e3843dff-d849-4187-a2ae-a65de3254375","Type":"ContainerStarted","Data":"a40de81bd5dcbefec07227c163947f3de428c9b1abb2404aa331fcd43cddcec5"}
Nov 02 20:03:06.476273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:06.475856    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:06.584570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:06.584467    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb" podStartSLOduration=1.6854240150000002 podStartE2EDuration="2.584454486s" totalImagesPullingTime="899.030471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:06.582772282 +0000 UTC m=+675.457043173" watchObservedRunningTime="2026-11-02 20:03:06.584454486 +0000 UTC m=+675.458725388"
Nov 02 20:03:12.452796 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:12.452759    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5fzvg"
Nov 02 20:03:12.453415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:12.452864    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-tm9g5"
Nov 02 20:03:16.092737 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:03:16.092700    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:03:17.485276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:17.485241    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-696cfc768f-hhwjb"
Nov 02 20:03:17.641994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:17.641959    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:03:17.642172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:17.642151    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" podUID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" containerName="manager" containerID="cri-o://8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b" gracePeriod=30
Nov 02 20:03:17.894753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:17.894725    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:03:18.045825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.045728    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\" (UniqueName: \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\") pod \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") "
Nov 02 20:03:18.045825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.045818    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\") pod \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\" (UID: \"f69e49d4-98eb-4b62-ae8a-80bfb53f2831\") "
Nov 02 20:03:18.047967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.047938    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4" pod "f69e49d4-98eb-4b62-ae8a-80bfb53f2831" (UID: "f69e49d4-98eb-4b62-ae8a-80bfb53f2831"). InnerVolumeSpecName "kube-api-access-t95h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:03:18.048104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.048049    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs" pod "f69e49d4-98eb-4b62-ae8a-80bfb53f2831" (UID: "f69e49d4-98eb-4b62-ae8a-80bfb53f2831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:03:18.147231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.147194    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95h4\" (UniqueName: \"kubernetes.io/projected/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-kube-api-access-t95h4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:18.147231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.147225    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f69e49d4-98eb-4b62-ae8a-80bfb53f2831-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:03:18.533981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.533949    5099 generic.go:356] "Generic (PLEG): container finished" podID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" containerID="8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b" exitCode=0
Nov 02 20:03:18.534389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.533992    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"
Nov 02 20:03:18.534389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.534034    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" event={"ID":"f69e49d4-98eb-4b62-ae8a-80bfb53f2831","Type":"ContainerDied","Data":"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"}
Nov 02 20:03:18.534389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.534087    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-8sr8j" event={"ID":"f69e49d4-98eb-4b62-ae8a-80bfb53f2831","Type":"ContainerDied","Data":"d2ab8d64adb69f25d86f1b074e3947eefb8708df470425df067e7e84b6bbed6d"}
Nov 02 20:03:18.534389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.534108    5099 scope.go:122] "RemoveContainer" containerID="8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"
Nov 02 20:03:18.550179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.550157    5099 scope.go:122] "RemoveContainer" containerID="8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"
Nov 02 20:03:18.550502 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:03:18.550481    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b\": container with ID starting with 8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b not found: ID does not exist" containerID="8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"
Nov 02 20:03:18.550549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.550511    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b"} err="failed to get container status \"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b\": rpc error: code = NotFound desc = could not find container \"8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b\": container with ID starting with 8872fdb73c43a4be686adb9cbd800f021c9890ffe010d95fc9a75a10a118a99b not found: ID does not exist"
Nov 02 20:03:18.687559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.687519    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:03:18.734790 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:18.734756    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-8sr8j"]
Nov 02 20:03:19.256132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:19.256104    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" path="/var/lib/kubelet/pods/f69e49d4-98eb-4b62-ae8a-80bfb53f2831/volumes"
Nov 02 20:03:29.115895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.115858    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:03:29.116385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.116278    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" containerName="manager"
Nov 02 20:03:29.116385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.116289    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" containerName="manager"
Nov 02 20:03:29.116385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.116383    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69e49d4-98eb-4b62-ae8a-80bfb53f2831" containerName="manager"
Nov 02 20:03:29.120697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.120678    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146543    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146581    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146607    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146693    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146801    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146820    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146843    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lm6\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.146967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146880    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.147031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.146977    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.219271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.219239    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:03:29.247721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247690    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247730    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247762    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247804    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247842    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247869    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.247917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247904    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lm6\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247949    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.247984    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.248178    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.248307    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.248414    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.248594    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.248669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.248649    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.250532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.250505    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.250892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.250872    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.301477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.301448    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.301698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.301674    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lm6\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\") pod \"maas-default-gateway-openshift-default-696f47c6c-wb4xd\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.433385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.433304    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:29.845103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.845076    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:03:29.846869 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:29.846846    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86df5e11_0409_4087_b289_899149ffab93.slice/crio-9a2ddda2acf2f17dc8b8a1aebb4e60020c61fd6ce81221376c4253f86466625e WatchSource:0}: Error finding container 9a2ddda2acf2f17dc8b8a1aebb4e60020c61fd6ce81221376c4253f86466625e: Status 404 returned error can't find the container with id 9a2ddda2acf2f17dc8b8a1aebb4e60020c61fd6ce81221376c4253f86466625e
Nov 02 20:03:29.849062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.849030    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:29.849194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.849089    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:29.849194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:29.849124    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:30.595934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:30.595899    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" event={"ID":"86df5e11-0409-4087-b289-899149ffab93","Type":"ContainerStarted","Data":"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"}
Nov 02 20:03:30.596366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:30.595953    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" event={"ID":"86df5e11-0409-4087-b289-899149ffab93","Type":"ContainerStarted","Data":"9a2ddda2acf2f17dc8b8a1aebb4e60020c61fd6ce81221376c4253f86466625e"}
Nov 02 20:03:30.814359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:30.814299    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podStartSLOduration=1.814285297 podStartE2EDuration="1.814285297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:30.765621766 +0000 UTC m=+699.639892654" watchObservedRunningTime="2026-11-02 20:03:30.814285297 +0000 UTC m=+699.688556184"
Nov 02 20:03:31.433891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:31.433856    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:31.435520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:31.435492    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:31.435639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:31.435549    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:32.433712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:32.433676    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:32.434150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:32.433748    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:33.434466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:33.434426    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:33.434901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:33.434489    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:34.434013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:34.433973    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:34.434267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:34.434031    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:35.434065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:35.434031    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:35.434490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:35.434096    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:36.434176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:36.434137    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-wb4xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 02 20:03:36.434584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:36.434195    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 02 20:03:38.085549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:38.085515    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:38.086068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:38.085899    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:38.086681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:38.086661    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:03:46.095569 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:03:46.095520    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:03:57.239264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.239185    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:03:57.243103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.243082    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.324975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.324941    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.324975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.324978    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.324997    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48xr\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325072    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325136    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325169    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325280    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325297    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.325411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.325389    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.341371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.341342    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9fcedd-openshift-default-dockercfg-rmhls\""
Nov 02 20:03:57.342705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.342676    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:03:57.426067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426031    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426092    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426108    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426124    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48xr\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426154    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426180    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426200    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426230    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426254    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426578    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426735    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426765    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.426992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.426955    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.427054 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.427021    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.428815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.428775    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.429229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.429208    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.480189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.480149    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.480341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.480264    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48xr\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.525906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.525828    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:03:57.530706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.530685    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.556834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.556805    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:57.582602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.582446    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ae8e6e-openshift-default-dockercfg-4p7q9\""
Nov 02 20:03:57.586386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.586356    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:03:57.628547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628505    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628560    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628623    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628667    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628705    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628730    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628758    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628816    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.628877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.628849    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhb54\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729622    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729673    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729704    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729729    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729759    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.729886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729832    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhb54\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729931    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.729958    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730020    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730488    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730559    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730589    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730723    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.730860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.730776    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.732405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.732382    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.732620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.732603    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.783994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.783919    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.784124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.784083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhb54\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.845060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.845011    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:57.980099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.980014    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:03:57.985149 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:57.985110    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e2ef40_1e0a_46a1_b942_1b0be8907ff1.slice/crio-5e167780a9eed634a9082addfe412dc02ecdb611762f6143d9c70f7098be4723 WatchSource:0}: Error finding container 5e167780a9eed634a9082addfe412dc02ecdb611762f6143d9c70f7098be4723: Status 404 returned error can't find the container with id 5e167780a9eed634a9082addfe412dc02ecdb611762f6143d9c70f7098be4723
Nov 02 20:03:57.990017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.989064    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:57.990017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.989136    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:57.990017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:57.989191    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:58.229555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.229519    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:03:58.232512 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:58.232467    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3473908_6129_4fcc_b8b5_b7424e8d4c67.slice/crio-41adfc7f8230a43fdf11d7740da6d818857fec8ca8f01fc0790c1bbbbe38b53d WatchSource:0}: Error finding container 41adfc7f8230a43fdf11d7740da6d818857fec8ca8f01fc0790c1bbbbe38b53d: Status 404 returned error can't find the container with id 41adfc7f8230a43fdf11d7740da6d818857fec8ca8f01fc0790c1bbbbe38b53d
Nov 02 20:03:58.244760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.244258    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:58.244760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.244338    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:58.244760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.244383    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:58.502277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.502242    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:03:58.506264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.506241    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.563407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.563311    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:03:58.618476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.618447    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-accdf5-openshift-default-dockercfg-c8k9m\""
Nov 02 20:03:58.653325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653287    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9fq\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653337    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653466    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653518    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653641    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653671    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653697    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653809    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.653906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.653875    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.736428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.736381    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerStarted","Data":"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"}
Nov 02 20:03:58.736623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.736455    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerStarted","Data":"5e167780a9eed634a9082addfe412dc02ecdb611762f6143d9c70f7098be4723"}
Nov 02 20:03:58.738173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.738139    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerStarted","Data":"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"}
Nov 02 20:03:58.738314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.738183    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerStarted","Data":"41adfc7f8230a43fdf11d7740da6d818857fec8ca8f01fc0790c1bbbbe38b53d"}
Nov 02 20:03:58.754902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.754832    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.754902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.754895    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.754943    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9fq\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.754982    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755024    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755057    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755146    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755171    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755199    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755658    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755665    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755699    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.755876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.755835    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.756160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.756025    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.758191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.758171    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.758503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.758475    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.796566 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.796536    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:03:58.800425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.800403    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.809252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.809231    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9fq\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.809377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.809237    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.822104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.822046    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:58.845940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.845910    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:58.850997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.850972    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-035e47-openshift-default-dockercfg-rx8rw\""
Nov 02 20:03:58.852870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.852823    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podStartSLOduration=1.852807781 podStartE2EDuration="1.852807781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:58.850427545 +0000 UTC m=+727.724698432" watchObservedRunningTime="2026-11-02 20:03:58.852807781 +0000 UTC m=+727.727078666"
Nov 02 20:03:58.857207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.857181    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:03:58.957916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.957873    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.957968    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958002    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958116    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958157    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958270    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958317    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958427    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:58.958515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:58.958491    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.059876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.059843    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.059876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.059880    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.059899    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.059963    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060005    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060096    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060142    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060201    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060258    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060343    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060497    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060543    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060599    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.060772 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.060754    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.062508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.062486    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.062681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.062661    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.074251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.074171    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podStartSLOduration=2.074155646 podStartE2EDuration="2.074155646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:59.072927308 +0000 UTC m=+727.947198197" watchObservedRunningTime="2026-11-02 20:03:59.074155646 +0000 UTC m=+727.948426533"
Nov 02 20:03:59.114077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.114043    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.114230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.114150    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.151821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.151774    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:03:59.212046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.212014    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:03:59.227227 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:59.227164    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b29987_1040_4f1b_aa93_715832dfd0a6.slice/crio-336a0aee444ea7d4d5c0fa14416c7ddbccfd6c8167c837fe9c5098c470f2e896 WatchSource:0}: Error finding container 336a0aee444ea7d4d5c0fa14416c7ddbccfd6c8167c837fe9c5098c470f2e896: Status 404 returned error can't find the container with id 336a0aee444ea7d4d5c0fa14416c7ddbccfd6c8167c837fe9c5098c470f2e896
Nov 02 20:03:59.295073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.289928    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.295073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.290015    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.295073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.290058    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.520716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.520689    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:03:59.522490 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:03:59.522454    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6369b245_aba9_46fc_b5dd_3f26e05e2b15.slice/crio-a6244bc46da443f3fbc71256d8376fb5ccf51df574822b1777eb8333dc04c32d WatchSource:0}: Error finding container a6244bc46da443f3fbc71256d8376fb5ccf51df574822b1777eb8333dc04c32d: Status 404 returned error can't find the container with id a6244bc46da443f3fbc71256d8376fb5ccf51df574822b1777eb8333dc04c32d
Nov 02 20:03:59.524993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.524965    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.525055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.525027    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.525093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.525054    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:03:59.557516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.557483    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:59.563750 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.563723    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:59.745589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.745545    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerStarted","Data":"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"}
Nov 02 20:03:59.746055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.745617    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerStarted","Data":"336a0aee444ea7d4d5c0fa14416c7ddbccfd6c8167c837fe9c5098c470f2e896"}
Nov 02 20:03:59.747824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.747793    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerStarted","Data":"289af6130ca8086588b591c492813f9abd640806f9a1d7c918ce51862092ed0b"}
Nov 02 20:03:59.747956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.747844    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerStarted","Data":"a6244bc46da443f3fbc71256d8376fb5ccf51df574822b1777eb8333dc04c32d"}
Nov 02 20:03:59.748183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.748165    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:59.749066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.749036    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:03:59.823122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.823042    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:03:59.850913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.850842    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:03:59.855842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.855773    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podStartSLOduration=1.8557614770000002 podStartE2EDuration="1.855761477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:59.854958971 +0000 UTC m=+728.729229895" watchObservedRunningTime="2026-11-02 20:03:59.855761477 +0000 UTC m=+728.730032344"
Nov 02 20:03:59.965038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:03:59.964982    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podStartSLOduration=1.9649691489999999 podStartE2EDuration="1.964969149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:59.964809391 +0000 UTC m=+728.839080279" watchObservedRunningTime="2026-11-02 20:03:59.964969149 +0000 UTC m=+728.839240036"
Nov 02 20:04:00.158807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.156847    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:00.184450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.184423    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:00.753820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.753766    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:00.754296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.754122    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:00.754970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.754947    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:00.755085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.755005    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:00.828052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:00.828018    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:01.759251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:01.759218    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:01.759894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:01.759878    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:12.063730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.063688    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:04:12.068765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.068747    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.113329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.113290    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7b7\" (UniqueName: \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.113493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.113358    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.121205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.121175    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:04:12.168162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.168124    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cc64d3-dockercfg-684pn\""
Nov 02 20:04:12.168355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.168223    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cc64d3\""
Nov 02 20:04:12.214453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.214418    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.214644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.214587    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7b7\" (UniqueName: \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.220197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.219712    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:04:12.222328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.222303    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.224919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.224896    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.271374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.271327    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:04:12.275873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.275839    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7b7\" (UniqueName: \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\") pod \"payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.382763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.382720    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.416206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.416160    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.416376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.416210    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwwc\" (UniqueName: \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.517292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.517255    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.517292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.517296    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwwc\" (UniqueName: \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.517988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.517967    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.571731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.571701    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwwc\" (UniqueName: \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\") pod \"payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.756471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.756444    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:04:12.758165 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:12.758141    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1aa0e1_9d3e_4523_9982_2b5a5dbdf21e.slice/crio-125dce78f3e562f1fc444b72d1ef0982e9bd50981569633efc9082756e92bdd6 WatchSource:0}: Error finding container 125dce78f3e562f1fc444b72d1ef0982e9bd50981569633efc9082756e92bdd6: Status 404 returned error can't find the container with id 125dce78f3e562f1fc444b72d1ef0982e9bd50981569633efc9082756e92bdd6
Nov 02 20:04:12.825189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.825150    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" event={"ID":"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e","Type":"ContainerStarted","Data":"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"}
Nov 02 20:04:12.825319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.825299    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:12.825367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.825318    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" event={"ID":"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e","Type":"ContainerStarted","Data":"125dce78f3e562f1fc444b72d1ef0982e9bd50981569633efc9082756e92bdd6"}
Nov 02 20:04:12.840193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.840168    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:12.935721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:12.935671    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" podStartSLOduration=0.935656441 podStartE2EDuration="935.656441ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:12.933217643 +0000 UTC m=+741.807488531" watchObservedRunningTime="2026-11-02 20:04:12.935656441 +0000 UTC m=+741.809927326"
Nov 02 20:04:13.207468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:13.207272    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:04:13.213474 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:13.213439    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed0e859_8ad2_46f7_b27e_2f30ab98e37d.slice/crio-13640f2aab9819b6c864686730fa22578dcb15b3b3a06f54abaf7b1f28aca371 WatchSource:0}: Error finding container 13640f2aab9819b6c864686730fa22578dcb15b3b3a06f54abaf7b1f28aca371: Status 404 returned error can't find the container with id 13640f2aab9819b6c864686730fa22578dcb15b3b3a06f54abaf7b1f28aca371
Nov 02 20:04:13.832671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:13.832627    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" event={"ID":"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d","Type":"ContainerStarted","Data":"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"}
Nov 02 20:04:13.832866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:13.832695    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" event={"ID":"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d","Type":"ContainerStarted","Data":"13640f2aab9819b6c864686730fa22578dcb15b3b3a06f54abaf7b1f28aca371"}
Nov 02 20:04:13.832866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:13.832812    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:13.942032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:13.941976    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" podStartSLOduration=1.941963004 podStartE2EDuration="1.941963004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:13.940043256 +0000 UTC m=+742.814314148" watchObservedRunningTime="2026-11-02 20:04:13.941963004 +0000 UTC m=+742.816233897"
Nov 02 20:04:16.093358 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:04:16.093322    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:04:22.192243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.192207    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:04:22.196234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.196207    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.249284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.249252    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:04:22.289806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.287260    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.289806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.287434    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scn4g\" (UniqueName: \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.306804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.293006    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2052e3\""
Nov 02 20:04:22.306804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.293041    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2052e3-dockercfg-bh9jx\""
Nov 02 20:04:22.345102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.345069    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:04:22.349070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.349044    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.388656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.388628    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scn4g\" (UniqueName: \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.388846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.388684    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.389378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.389359    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.401901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.401875    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:04:22.442850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.442763    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scn4g\" (UniqueName: \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\") pod \"payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.489675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.489639    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.489897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.489768    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdw9x\" (UniqueName: \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.511514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.511480    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:22.591470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.591435    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.591625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.591505    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdw9x\" (UniqueName: \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.592406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.592377    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.645326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.645299    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdw9x\" (UniqueName: \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\") pod \"payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.663262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.663236    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:22.879931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:22.879907    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:04:22.881235 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:22.881207    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4b5e21_b3da_45d3_8367_a5ade1342651.slice/crio-3aad548c541ef757f1200122f11fc15435495bf5afb8418dbe0197e867006788 WatchSource:0}: Error finding container 3aad548c541ef757f1200122f11fc15435495bf5afb8418dbe0197e867006788: Status 404 returned error can't find the container with id 3aad548c541ef757f1200122f11fc15435495bf5afb8418dbe0197e867006788
Nov 02 20:04:23.031765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.031732    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:04:23.032945 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:23.032917    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5f3706_04b1_4f15_beaf_c82e37393866.slice/crio-72585df5460ae8d2784b104fdfad9817a253aebbdb0169195d82c9e5fc6ec9a1 WatchSource:0}: Error finding container 72585df5460ae8d2784b104fdfad9817a253aebbdb0169195d82c9e5fc6ec9a1: Status 404 returned error can't find the container with id 72585df5460ae8d2784b104fdfad9817a253aebbdb0169195d82c9e5fc6ec9a1
Nov 02 20:04:23.836812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.836765    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:04:23.887700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.887666    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" event={"ID":"3a5f3706-04b1-4f15-beaf-c82e37393866","Type":"ContainerStarted","Data":"d15639f5eb4c7111c6666e8426f397c8984a894aeb5eea230307d851f99a47b6"}
Nov 02 20:04:23.887899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.887732    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:23.887899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.887756    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" event={"ID":"3a5f3706-04b1-4f15-beaf-c82e37393866","Type":"ContainerStarted","Data":"72585df5460ae8d2784b104fdfad9817a253aebbdb0169195d82c9e5fc6ec9a1"}
Nov 02 20:04:23.889134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.889110    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" event={"ID":"9f4b5e21-b3da-45d3-8367-a5ade1342651","Type":"ContainerStarted","Data":"f11cad343e237e36a8cfd28988a1ad82cd13b1d8c626f1a343b22f1ace0158e1"}
Nov 02 20:04:23.889263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.889149    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" event={"ID":"9f4b5e21-b3da-45d3-8367-a5ade1342651","Type":"ContainerStarted","Data":"3aad548c541ef757f1200122f11fc15435495bf5afb8418dbe0197e867006788"}
Nov 02 20:04:23.889263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:23.889190    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:24.057301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.057225    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" podStartSLOduration=2.057204365 podStartE2EDuration="2.057204365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:24.052147618 +0000 UTC m=+752.926418507" watchObservedRunningTime="2026-11-02 20:04:24.057204365 +0000 UTC m=+752.931475254"
Nov 02 20:04:24.159211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.159153    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" podStartSLOduration=2.159137774 podStartE2EDuration="2.159137774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:24.158824076 +0000 UTC m=+753.033094966" watchObservedRunningTime="2026-11-02 20:04:24.159137774 +0000 UTC m=+753.033408662"
Nov 02 20:04:24.795837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.795803    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:04:24.799648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.799628    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:24.841568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.841543    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:04:24.849496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.849466    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ae8e6e\""
Nov 02 20:04:24.851884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.851860    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:04:24.914176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.909905    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ae8e6e-dockercfg-r8d9h\""
Nov 02 20:04:24.921526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.921490    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:24.921929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.921910    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22rc\" (UniqueName: \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:24.948368 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.948330    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:04:24.952107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:24.952089    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.005015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.004973    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:04:25.024186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.024158    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.024359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.024232    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22rc\" (UniqueName: \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.025484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.025459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.078024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.077946    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22rc\" (UniqueName: \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\") pod \"payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.113839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.113797    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.125511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.125381    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.125589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.125506    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q7v\" (UniqueName: \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.226504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.226472    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q7v\" (UniqueName: \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.226677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.226595    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.227342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.227321    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.280622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.280590    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q7v\" (UniqueName: \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\") pod \"payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.481703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.481673    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:04:25.483185 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:25.483157    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf2296a_806b_4344_8d11_d4c8d9f42783.slice/crio-e2d3a803d18de343171983676ec492359fb0026303cfe7939818024c9975d4af WatchSource:0}: Error finding container e2d3a803d18de343171983676ec492359fb0026303cfe7939818024c9975d4af: Status 404 returned error can't find the container with id e2d3a803d18de343171983676ec492359fb0026303cfe7939818024c9975d4af
Nov 02 20:04:25.565648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.565624    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:25.929078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.929042    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" event={"ID":"aaf2296a-806b-4344-8d11-d4c8d9f42783","Type":"ContainerStarted","Data":"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"}
Nov 02 20:04:25.929479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.929105    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" event={"ID":"aaf2296a-806b-4344-8d11-d4c8d9f42783","Type":"ContainerStarted","Data":"e2d3a803d18de343171983676ec492359fb0026303cfe7939818024c9975d4af"}
Nov 02 20:04:25.929479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.929137    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:25.931198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:25.931175    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:04:25.932898 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:25.932876    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode788066f_f2a1_4b06_9b2e_955d313f6367.slice/crio-ba21a6be3e54cb0a7c75de9874615efa6a9c0f24a58b4ba02aa9474b6a12e729 WatchSource:0}: Error finding container ba21a6be3e54cb0a7c75de9874615efa6a9c0f24a58b4ba02aa9474b6a12e729: Status 404 returned error can't find the container with id ba21a6be3e54cb0a7c75de9874615efa6a9c0f24a58b4ba02aa9474b6a12e729
Nov 02 20:04:26.036188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:26.036143    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" podStartSLOduration=2.036130008 podStartE2EDuration="2.036130008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:26.036018536 +0000 UTC m=+754.910289423" watchObservedRunningTime="2026-11-02 20:04:26.036130008 +0000 UTC m=+754.910400895"
Nov 02 20:04:26.936858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:26.936817    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" event={"ID":"e788066f-f2a1-4b06-9b2e-955d313f6367","Type":"ContainerStarted","Data":"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"}
Nov 02 20:04:26.937354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:26.937086    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:26.937354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:26.937121    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" event={"ID":"e788066f-f2a1-4b06-9b2e-955d313f6367","Type":"ContainerStarted","Data":"ba21a6be3e54cb0a7c75de9874615efa6a9c0f24a58b4ba02aa9474b6a12e729"}
Nov 02 20:04:27.044348 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:27.044283    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" podStartSLOduration=3.044271293 podStartE2EDuration="3.044271293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:27.043778678 +0000 UTC m=+755.918049563" watchObservedRunningTime="2026-11-02 20:04:27.044271293 +0000 UTC m=+755.918542180"
Nov 02 20:04:29.695569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.695451    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:04:29.700479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.700452    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.752415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.752369    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:04:29.784320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.784279    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxbx\" (UniqueName: \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.784514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.784491    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.795960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.795930    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-accdf5\""
Nov 02 20:04:29.796112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.796037    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-accdf5-dockercfg-hvk8r\""
Nov 02 20:04:29.845843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.845805    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:04:29.849769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.849752    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:29.889322 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.889288    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.889502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.889389    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxbx\" (UniqueName: \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.890606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.890577    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.914094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.914060    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:04:29.948641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.948558    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxbx\" (UniqueName: \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\") pod \"payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:29.990389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.990355    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdlf\" (UniqueName: \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:29.990540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:29.990397    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.015068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.015039    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:30.091688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.091641    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdlf\" (UniqueName: \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.091824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.091706    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.092829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.092777    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.149002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.148969    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdlf\" (UniqueName: \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\") pod \"payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.194943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.194897    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:30.393942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.393906    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:04:30.401062 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:30.401034    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8065982f_c65d_489d_ac49_017ed5f79b9d.slice/crio-f31d6cec254fab65a38d6f225d532a55450e3b58fbdfec03f5b89712c6c6ce48 WatchSource:0}: Error finding container f31d6cec254fab65a38d6f225d532a55450e3b58fbdfec03f5b89712c6c6ce48: Status 404 returned error can't find the container with id f31d6cec254fab65a38d6f225d532a55450e3b58fbdfec03f5b89712c6c6ce48
Nov 02 20:04:30.652756 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.652730    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:04:30.654298 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:04:30.654266    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccf63bf_fc64_4d4e_b4b4_6d028b76fa46.slice/crio-a6b8fa885cbc1658ba0a19fcc409641352b02a69fde445d298ed1c7139df18d5 WatchSource:0}: Error finding container a6b8fa885cbc1658ba0a19fcc409641352b02a69fde445d298ed1c7139df18d5: Status 404 returned error can't find the container with id a6b8fa885cbc1658ba0a19fcc409641352b02a69fde445d298ed1c7139df18d5
Nov 02 20:04:30.984176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.983110    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" event={"ID":"8065982f-c65d-489d-ac49-017ed5f79b9d","Type":"ContainerStarted","Data":"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"}
Nov 02 20:04:30.984176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.983201    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" event={"ID":"8065982f-c65d-489d-ac49-017ed5f79b9d","Type":"ContainerStarted","Data":"f31d6cec254fab65a38d6f225d532a55450e3b58fbdfec03f5b89712c6c6ce48"}
Nov 02 20:04:30.984176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.983868    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:30.991183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.991140    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" event={"ID":"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46","Type":"ContainerStarted","Data":"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"}
Nov 02 20:04:30.991396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.991227    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" event={"ID":"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46","Type":"ContainerStarted","Data":"a6b8fa885cbc1658ba0a19fcc409641352b02a69fde445d298ed1c7139df18d5"}
Nov 02 20:04:30.991634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:30.991588    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:31.093958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:31.093900    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" podStartSLOduration=2.093884792 podStartE2EDuration="2.093884792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:31.092176136 +0000 UTC m=+759.966447035" watchObservedRunningTime="2026-11-02 20:04:31.093884792 +0000 UTC m=+759.968155680"
Nov 02 20:04:31.206807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:31.205957    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" podStartSLOduration=2.205937795 podStartE2EDuration="2.205937795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:31.203466453 +0000 UTC m=+760.077737343" watchObservedRunningTime="2026-11-02 20:04:31.205937795 +0000 UTC m=+760.080208685"
Nov 02 20:04:34.923874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:34.923846    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:04:34.926151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:34.926130    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:04:36.941509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:36.941481    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:04:37.946996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:37.946966    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:04:40.060749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.060718    5099 generic.go:356] "Generic (PLEG): container finished" podID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerID="96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822" exitCode=0
Nov 02 20:04:40.062227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.060798    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerDied","Data":"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"}
Nov 02 20:04:40.062227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.061176    5099 scope.go:122] "RemoveContainer" containerID="96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"
Nov 02 20:04:40.062748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.062389    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:40.062748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.062450    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:40.062748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:40.062481    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:41.075856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.075805    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerStarted","Data":"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"}
Nov 02 20:04:41.078693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.078665    5099 generic.go:356] "Generic (PLEG): container finished" podID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerID="289af6130ca8086588b591c492813f9abd640806f9a1d7c918ce51862092ed0b" exitCode=0
Nov 02 20:04:41.078864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.078740    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerDied","Data":"289af6130ca8086588b591c492813f9abd640806f9a1d7c918ce51862092ed0b"}
Nov 02 20:04:41.079184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.079167    5099 scope.go:122] "RemoveContainer" containerID="289af6130ca8086588b591c492813f9abd640806f9a1d7c918ce51862092ed0b"
Nov 02 20:04:41.080538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.080512    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:41.080650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.080589    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:41.080650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.080639    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:41.847952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.847852    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:41.852248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.851912    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 20:04:41.852248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:41.851972    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 20:04:42.010003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.009976    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:04:42.010879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.010859    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:04:42.093741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.093706    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerStarted","Data":"177e5ba5e2a1f23719b098ba161f49351b42e408189a1de9683731d06ca45ba6"}
Nov 02 20:04:42.152932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.152903    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:42.153313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.153291    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:42.153468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.153447    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:42.850313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.850273    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 20:04:42.850660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:42.850614    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 20:04:43.157806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:43.152209    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:43.157806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:43.152267    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:43.848682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:43.848195    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 20:04:43.848682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:43.848277    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 20:04:44.154351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:44.154313    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:44.154680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:44.154644    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:44.848194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:44.848154    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 20:04:44.848811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:44.848752    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 20:04:45.162808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.159041    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:45.163309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.163278    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:45.755807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.753891    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:45.755807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.754855    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:45.853808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.849222    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 20:04:45.853808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:45.849299    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 20:04:46.134313 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:04:46.134276    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:04:46.165275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:46.165239    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:46.165570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:46.165543    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:46.871806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:46.870762    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:46.882133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:46.882056    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:04:47.157314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:47.157207    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 20:04:47.157685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:47.157561    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 20:04:48.168117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:48.168078    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:48.170051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:48.170024    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:04:52.259549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.259512    5099 generic.go:356] "Generic (PLEG): container finished" podID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerID="3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00" exitCode=0
Nov 02 20:04:52.260163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.259569    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerDied","Data":"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"}
Nov 02 20:04:52.260163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.260061    5099 scope.go:122] "RemoveContainer" containerID="3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"
Nov 02 20:04:52.262367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.262337    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:52.262474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.262440    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:52.262519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:52.262492    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:04:53.298852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:53.298817    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerStarted","Data":"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"}
Nov 02 20:04:53.822767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:53.822732    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:53.822971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:53.822904    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:04:53.822971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:53.822957    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 20:04:54.823389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:54.823351    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:04:54.823817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:54.823406    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 20:04:55.823440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:55.823405    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:04:55.823893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:55.823469    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 20:04:56.822493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:56.822458    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:04:56.822695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:56.822625    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 20:04:57.823121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:57.823089    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 20:04:57.823534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:57.823143    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 20:04:58.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:58.826731    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:58.827178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:58.827161    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:04:58.827996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:04:58.827974    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:05:00.789028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:00.788979    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:05:00.789557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:00.789067    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:05:02.354673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.354581    5099 generic.go:356] "Generic (PLEG): container finished" podID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerID="f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913" exitCode=0
Nov 02 20:05:02.354673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.354651    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerDied","Data":"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"}
Nov 02 20:05:02.355131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.355068    5099 scope.go:122] "RemoveContainer" containerID="f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"
Nov 02 20:05:02.356323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.356295    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:02.356407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.356365    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:02.356407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:02.356397    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:05:03.367992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:03.367958    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerStarted","Data":"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"}
Nov 02 20:05:03.560219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:03.560183    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:05:03.560558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:03.560537    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 20:05:03.560706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:03.560678    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 20:05:04.560034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:04.559995    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 20:05:04.560629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:04.560596    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 20:05:05.558757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:05.558258    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 20:05:05.558757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:05.558344    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 20:05:06.561215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:06.561176    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 20:05:06.561851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:06.561814    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 20:05:07.561807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:07.561203    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 20:05:07.561807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:07.561266    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 20:05:08.565184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:08.565143    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:05:08.565986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:08.565965    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:05:08.566604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:05:08.566586    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:05:16.094681 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:05:16.094643    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:05:46.096435 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:05:46.096396    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:06:09.780194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.780163    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:06:09.785041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.785013    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.837362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.837332    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:06:09.881557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.881518    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"openshift-service-ca.crt\""
Nov 02 20:06:09.881557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.881533    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"kube-root-ca.crt\""
Nov 02 20:06:09.881748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.881597    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"default-dockercfg-qzq75\""
Nov 02 20:06:09.881982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.881967    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"premium-w6-982b8a-kserve-self-signed-certs\""
Nov 02 20:06:09.900478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900443    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtzr\" (UniqueName: \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.900550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900494    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.900586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900558    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.900586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900579    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.900687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900667    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:09.900855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:09.900840    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.002362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.002320    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.002542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.002423    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.002542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.002462    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.002802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.002741    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.002896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.002778    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.003113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.003080    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.003243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.003152    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtzr\" (UniqueName: \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.003537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.003517    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.003914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.003892    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.004901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.004876    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.005154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.005128    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.065105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.065020    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtzr\" (UniqueName: \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\") pod \"premium-w6-982b8a-kserve-7479576cf8-xl5vd\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") " pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.116062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.116033    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:10.523002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.522968    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:06:10.524445 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:06:10.524418    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4804654_676a_4a35_9c20_8761ec9324e8.slice/crio-87557173afb3fdcb90220092c1c6c78378f0000cc0247f4385122786c2177d28 WatchSource:0}: Error finding container 87557173afb3fdcb90220092c1c6c78378f0000cc0247f4385122786c2177d28: Status 404 returned error can't find the container with id 87557173afb3fdcb90220092c1c6c78378f0000cc0247f4385122786c2177d28
Nov 02 20:06:10.795702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.795620    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" event={"ID":"a4804654-676a-4a35-9c20-8761ec9324e8","Type":"ContainerStarted","Data":"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"}
Nov 02 20:06:10.795702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.795667    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" event={"ID":"a4804654-676a-4a35-9c20-8761ec9324e8","Type":"ContainerStarted","Data":"87557173afb3fdcb90220092c1c6c78378f0000cc0247f4385122786c2177d28"}
Nov 02 20:06:10.906260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:10.906210    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podStartSLOduration=1.906197418 podStartE2EDuration="1.906197418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:10.904813946 +0000 UTC m=+859.779084836" watchObservedRunningTime="2026-11-02 20:06:10.906197418 +0000 UTC m=+859.780468305"
Nov 02 20:06:11.116434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:11.116403    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:16.092940 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:06:16.092904    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:06:20.117029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:20.116998    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:20.129466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:20.129442    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:20.864476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:20.864444    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:06:46.093636 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:06:46.093602    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:06:51.365291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:51.365261    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:06:51.366575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:06:51.366553    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:07:06.904722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.904686    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:07:06.909350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.909326    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933267    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933391    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933418    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbx5\" (UniqueName: \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933589    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933652    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.933748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.933734    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:06.962133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:06.962096    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:07:07.005259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.005226    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"openshift-service-ca.crt\""
Nov 02 20:07:07.005433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.005288    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"default-dockercfg-94w94\""
Nov 02 20:07:07.005433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.005292    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"kube-root-ca.crt\""
Nov 02 20:07:07.006127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.006108    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"premium-w3-2052e3-kserve-self-signed-certs\""
Nov 02 20:07:07.034670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.034634    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.034868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.034708    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.034868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.034743    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.044015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.034883    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.044015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.035263    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.044015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.035344    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.044015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.035722    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbx5\" (UniqueName: \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.049341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.049268    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.050644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.050620    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.055483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.055424    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.063407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.063372    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.105907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.105875    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbx5\" (UniqueName: \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\") pod \"premium-w3-2052e3-kserve-65bd69f764-tb4nt\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") " pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.221267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.221190    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:07.628772 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:07.628747    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:07:07.630191 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:07:07.630161    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b47d5a_d799_4f7f_ad0b_a96cf6d6ec05.slice/crio-957cd778b048ebab3371002c5a7e88754e9dfb971fb1384119da11bb33432fc0 WatchSource:0}: Error finding container 957cd778b048ebab3371002c5a7e88754e9dfb971fb1384119da11bb33432fc0: Status 404 returned error can't find the container with id 957cd778b048ebab3371002c5a7e88754e9dfb971fb1384119da11bb33432fc0
Nov 02 20:07:08.155503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:08.155464    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" event={"ID":"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05","Type":"ContainerStarted","Data":"79d0d9a7b558f0b46a7a59052c7728ed6e511401c4e18d2666d28362b9a26e86"}
Nov 02 20:07:08.156002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:08.155512    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" event={"ID":"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05","Type":"ContainerStarted","Data":"957cd778b048ebab3371002c5a7e88754e9dfb971fb1384119da11bb33432fc0"}
Nov 02 20:07:08.221330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:08.221296    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:08.265689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:08.265640    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podStartSLOduration=2.265627544 podStartE2EDuration="2.265627544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:08.263208379 +0000 UTC m=+917.137479270" watchObservedRunningTime="2026-11-02 20:07:08.265627544 +0000 UTC m=+917.139898428"
Nov 02 20:07:16.094140 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:07:16.094100    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:07:17.221704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:17.221675    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:17.235103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:17.235073    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:18.224162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:18.224129    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:07:30.698503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.698430    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:07:30.702753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.702734    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.752377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.752338    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2052e3-openshift-default-dockercfg-8r7gl\""
Nov 02 20:07:30.817410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.817369    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.817745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.817726    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.817951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.817934    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.818113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.818094    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qbhn\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.823902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.823870    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.824139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.824118    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.825381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.825360    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.825590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.825573    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.825839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.825822    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.831747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.831724    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:07:30.926840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.926806    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.926856    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.926996    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927048    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qbhn\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927075    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927121    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927167    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927215    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927296    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927312    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927553    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927615    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927820    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.927939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.927912    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.929696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.929667    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.930013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.929993    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.981213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.981145    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:30.981501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:30.981474    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qbhn\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:31.016206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:31.016167    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:31.437302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:31.437274    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:07:31.438893 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:07:31.438860    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a701bd_7a2c_4016_a17c_945e6208040f.slice/crio-6dc576a42a08d65b6f276338f533334935ffd844877662a958f4c63b2e0afa96 WatchSource:0}: Error finding container 6dc576a42a08d65b6f276338f533334935ffd844877662a958f4c63b2e0afa96: Status 404 returned error can't find the container with id 6dc576a42a08d65b6f276338f533334935ffd844877662a958f4c63b2e0afa96
Nov 02 20:07:31.441627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:31.441584    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:31.441742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:31.441671    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:31.441742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:31.441714    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:07:32.313813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:32.313747    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" event={"ID":"39a701bd-7a2c-4016-a17c-945e6208040f","Type":"ContainerStarted","Data":"4f260fa5b99e377caf79cae122be6d1036e43f3e3ff8310ed5b20856974a93eb"}
Nov 02 20:07:32.314206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:32.313821    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" event={"ID":"39a701bd-7a2c-4016-a17c-945e6208040f","Type":"ContainerStarted","Data":"6dc576a42a08d65b6f276338f533334935ffd844877662a958f4c63b2e0afa96"}
Nov 02 20:07:32.426200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:32.426150    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podStartSLOduration=2.4261341659999998 podStartE2EDuration="2.426134166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:32.423911111 +0000 UTC m=+941.298182000" watchObservedRunningTime="2026-11-02 20:07:32.426134166 +0000 UTC m=+941.300405053"
Nov 02 20:07:33.016693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:33.016661    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:33.018003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:33.017981    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 20:07:33.018082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:33.018030    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 20:07:34.016661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:34.016629    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 20:07:34.017065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:34.016680    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 20:07:35.017481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:35.017444    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 20:07:35.017905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:35.017503    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 20:07:36.016665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:36.016628    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 20:07:36.016866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:36.016685    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 20:07:37.017092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:37.017063    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 20:07:37.017490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:37.017144    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 20:07:38.088591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:38.088555    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:38.089119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:38.088987    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:38.089753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:38.089734    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:07:46.094353 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:07:46.094314    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:07:57.818609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.818573    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"]
Nov 02 20:07:57.824091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.824071    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:57.875083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.875054    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"]
Nov 02 20:07:57.918605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.918544    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:07:57.918801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.918731    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:07:57.969754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.969716    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:57.969957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.969863    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:57.969957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:57.969902    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt78q\" (UniqueName: \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.071015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.070930    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.071015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.070997    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.071237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.071106    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt78q\" (UniqueName: \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.071582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.071562    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.073318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.073296    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.125206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.125171    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt78q\" (UniqueName: \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\") pod \"maas-api-revoke-sub-a7b8aaee-d8nfh\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.137452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.137424    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:07:58.506427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.506400    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"]
Nov 02 20:07:58.507680 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:07:58.507656    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79caf93d_a4e3_4a4d_a733_b223d2f6ec4a.slice/crio-bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db WatchSource:0}: Error finding container bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db: Status 404 returned error can't find the container with id bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db
Nov 02 20:07:58.510471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:58.510455    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:07:59.501410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:07:59.501378    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh" event={"ID":"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a","Type":"ContainerStarted","Data":"bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db"}
Nov 02 20:08:03.530004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:03.529964    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh" event={"ID":"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a","Type":"ContainerStarted","Data":"c42ce5b91b56d504b9c218c91f1996b4c11b201dd390ed06259d63af24f9679d"}
Nov 02 20:08:03.640799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:03.640728    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh" podStartSLOduration=2.468972366 podStartE2EDuration="6.640708382s" totalImagesPullingTime="4.171736016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:03.637052906 +0000 UTC m=+972.511323793" watchObservedRunningTime="2026-11-02 20:08:03.640708382 +0000 UTC m=+972.514979272"
Nov 02 20:08:04.537157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:04.537118    5099 generic.go:356] "Generic (PLEG): container finished" podID="79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" containerID="c42ce5b91b56d504b9c218c91f1996b4c11b201dd390ed06259d63af24f9679d" exitCode=0
Nov 02 20:08:04.537614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:04.537199    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh" event={"ID":"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a","Type":"ContainerDied","Data":"c42ce5b91b56d504b9c218c91f1996b4c11b201dd390ed06259d63af24f9679d"}
Nov 02 20:08:05.694121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.694097    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:08:05.768621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.768592    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\") pod \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") "
Nov 02 20:08:05.768819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.768649    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\") pod \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") "
Nov 02 20:08:05.768819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.768696    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\" (UniqueName: \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\") pod \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\" (UID: \"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a\") "
Nov 02 20:08:05.768996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.768972    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca" pod "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" (UID: "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:05.771065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.771035    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q" pod "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" (UID: "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a"). InnerVolumeSpecName "kube-api-access-tt78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:05.771336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.771314    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca" pod "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" (UID: "79caf93d-a4e3-4a4d-a733-b223d2f6ec4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:05.872938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.872904    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:05.872938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.872934    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:05.872938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:05.872945    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt78q\" (UniqueName: \"kubernetes.io/projected/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a-kube-api-access-tt78q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:06.552016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:06.551987    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"
Nov 02 20:08:06.552199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:06.551980    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh" event={"ID":"79caf93d-a4e3-4a4d-a733-b223d2f6ec4a","Type":"ContainerDied","Data":"bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db"}
Nov 02 20:08:06.552199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:06.552126    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc32fd14e1a532867a80bdca355f8d945797fd385361966c5948127cfeac12db"
Nov 02 20:08:15.712341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.712293    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"]
Nov 02 20:08:15.712864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.712848    5099 cpu_manager.go:414] "containerMap: removing container" podUID="79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" containerName="revoke-keys"
Nov 02 20:08:15.712864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.712864    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" containerName="revoke-keys"
Nov 02 20:08:15.712975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.712966    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" containerName="revoke-keys"
Nov 02 20:08:15.716227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.716206    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.769658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.769626    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"]
Nov 02 20:08:15.789625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.789595    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrfd\" (UniqueName: \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.789808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.789632    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.789872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.789852    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.813286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.813262    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:15.813453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.813302    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:15.891007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.890968    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.891169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.891058    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrfd\" (UniqueName: \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.891169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.891091    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.891725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.891706    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.893591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.893566    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:15.945975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:15.945950    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrfd\" (UniqueName: \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\") pod \"maas-api-revoke-sub-d53f9f72-lbxfj\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:16.030540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:16.030448    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:16.113033 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:08:16.112846    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:08:16.426433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:16.426404    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"]
Nov 02 20:08:16.427416 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:16.427389    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f6a568_5155_4ba7_bd67_2e929d9006fb.slice/crio-dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4 WatchSource:0}: Error finding container dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4: Status 404 returned error can't find the container with id dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4
Nov 02 20:08:16.611732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:16.611700    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj" event={"ID":"50f6a568-5155-4ba7-bd67-2e929d9006fb","Type":"ContainerStarted","Data":"dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4"}
Nov 02 20:08:17.618706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:17.618672    5099 generic.go:356] "Generic (PLEG): container finished" podID="50f6a568-5155-4ba7-bd67-2e929d9006fb" containerID="3d9601125310eae0b865b131a212dbf046244d2f46eb19b6afc0a61308a2849c" exitCode=0
Nov 02 20:08:17.619140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:17.618759    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj" event={"ID":"50f6a568-5155-4ba7-bd67-2e929d9006fb","Type":"ContainerDied","Data":"3d9601125310eae0b865b131a212dbf046244d2f46eb19b6afc0a61308a2849c"}
Nov 02 20:08:18.781079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.781047    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:18.922926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.922838    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\") pod \"50f6a568-5155-4ba7-bd67-2e929d9006fb\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") "
Nov 02 20:08:18.923185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.923030    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\" (UniqueName: \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\") pod \"50f6a568-5155-4ba7-bd67-2e929d9006fb\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") "
Nov 02 20:08:18.923185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.923097    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\") pod \"50f6a568-5155-4ba7-bd67-2e929d9006fb\" (UID: \"50f6a568-5155-4ba7-bd67-2e929d9006fb\") "
Nov 02 20:08:18.923367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.923260    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca" pod "50f6a568-5155-4ba7-bd67-2e929d9006fb" (UID: "50f6a568-5155-4ba7-bd67-2e929d9006fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:18.923561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.923536    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:18.925666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.925632    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd" pod "50f6a568-5155-4ba7-bd67-2e929d9006fb" (UID: "50f6a568-5155-4ba7-bd67-2e929d9006fb"). InnerVolumeSpecName "kube-api-access-mcrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:18.925776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:18.925691    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca" pod "50f6a568-5155-4ba7-bd67-2e929d9006fb" (UID: "50f6a568-5155-4ba7-bd67-2e929d9006fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:19.024320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:19.024285    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrfd\" (UniqueName: \"kubernetes.io/projected/50f6a568-5155-4ba7-bd67-2e929d9006fb-kube-api-access-mcrfd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:19.024320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:19.024315    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50f6a568-5155-4ba7-bd67-2e929d9006fb-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:19.631760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:19.631727    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"
Nov 02 20:08:19.631963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:19.631755    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj" event={"ID":"50f6a568-5155-4ba7-bd67-2e929d9006fb","Type":"ContainerDied","Data":"dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4"}
Nov 02 20:08:19.631963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:19.631835    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2a01b75487ae14342fa1f03c5dfe02fff742bf22b687cf222a5d3be945f5e4"
Nov 02 20:08:20.784285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.784244    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"]
Nov 02 20:08:20.785022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.784999    5099 cpu_manager.go:414] "containerMap: removing container" podUID="50f6a568-5155-4ba7-bd67-2e929d9006fb" containerName="revoke-keys"
Nov 02 20:08:20.785134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.785023    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f6a568-5155-4ba7-bd67-2e929d9006fb" containerName="revoke-keys"
Nov 02 20:08:20.785216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.785202    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f6a568-5155-4ba7-bd67-2e929d9006fb" containerName="revoke-keys"
Nov 02 20:08:20.788864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.788844    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:20.840951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.840918    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"]
Nov 02 20:08:20.884319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.884289    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:20.884497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.884359    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:20.947629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.947598    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:20.947827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.947659    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4rr\" (UniqueName: \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:20.947827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:20.947744    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.049336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.049239    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4rr\" (UniqueName: \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.049336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.049282    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.049549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.049433    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.050107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.050083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.051807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.051758    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.110828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.110772    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4rr\" (UniqueName: \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\") pod \"maas-api-revoke-sub-2a83434b-hq4cq\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.404016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.403986    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:21.768519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:21.768491    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"]
Nov 02 20:08:21.769546 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:21.769515    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4eda95_3565_4af4_8d05_695b586a1fac.slice/crio-066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc WatchSource:0}: Error finding container 066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc: Status 404 returned error can't find the container with id 066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc
Nov 02 20:08:22.652846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:22.652811    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq" event={"ID":"7d4eda95-3565-4af4-8d05-695b586a1fac","Type":"ContainerStarted","Data":"85ee8f2df415f52c68fd9a334bdf85a650ff8334171c01838b03c56e36f17b90"}
Nov 02 20:08:22.653229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:22.652868    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq" event={"ID":"7d4eda95-3565-4af4-8d05-695b586a1fac","Type":"ContainerStarted","Data":"066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc"}
Nov 02 20:08:22.760576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:22.760530    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq" podStartSLOduration=2.760516485 podStartE2EDuration="2.760516485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:22.759431196 +0000 UTC m=+991.633702094" watchObservedRunningTime="2026-11-02 20:08:22.760516485 +0000 UTC m=+991.634787372"
Nov 02 20:08:23.186803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.186759    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"]
Nov 02 20:08:23.190650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.190627    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.244774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.244739    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"]
Nov 02 20:08:23.374287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.374249    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.374525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.374496    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.374611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.374570    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht7q\" (UniqueName: \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.476049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.475953    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.476049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.476019    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht7q\" (UniqueName: \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.476284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.476074    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.476659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.476635    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.478389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.478371    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.530222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.530195    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht7q\" (UniqueName: \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\") pod \"maas-api-revoke-sub-ce6317f9-7498v\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:23.661623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.661583    5099 generic.go:356] "Generic (PLEG): container finished" podID="7d4eda95-3565-4af4-8d05-695b586a1fac" containerID="85ee8f2df415f52c68fd9a334bdf85a650ff8334171c01838b03c56e36f17b90" exitCode=0
Nov 02 20:08:23.662046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.661659    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq" event={"ID":"7d4eda95-3565-4af4-8d05-695b586a1fac","Type":"ContainerDied","Data":"85ee8f2df415f52c68fd9a334bdf85a650ff8334171c01838b03c56e36f17b90"}
Nov 02 20:08:23.804554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:23.804469    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:24.174922 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:24.174886    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cade08b_806e_4545_99a8_188f8c78d0ba.slice/crio-a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828 WatchSource:0}: Error finding container a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828: Status 404 returned error can't find the container with id a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828
Nov 02 20:08:24.175605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.175572    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"]
Nov 02 20:08:24.669868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.669821    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v" event={"ID":"7cade08b-806e-4545-99a8-188f8c78d0ba","Type":"ContainerStarted","Data":"a31ad549decaf8b467881ef9a0c9b5873006c404923692a669dac8c86444104b"}
Nov 02 20:08:24.670303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.669899    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v" event={"ID":"7cade08b-806e-4545-99a8-188f8c78d0ba","Type":"ContainerStarted","Data":"a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828"}
Nov 02 20:08:24.781350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.781273    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v" podStartSLOduration=1.781252576 podStartE2EDuration="1.781252576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:24.77914512 +0000 UTC m=+993.653416010" watchObservedRunningTime="2026-11-02 20:08:24.781252576 +0000 UTC m=+993.655523467"
Nov 02 20:08:24.821143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.821115    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:24.995347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.995263    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\") pod \"7d4eda95-3565-4af4-8d05-695b586a1fac\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") "
Nov 02 20:08:24.995489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.995465    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\" (UniqueName: \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\") pod \"7d4eda95-3565-4af4-8d05-695b586a1fac\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") "
Nov 02 20:08:24.995532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.995497    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\") pod \"7d4eda95-3565-4af4-8d05-695b586a1fac\" (UID: \"7d4eda95-3565-4af4-8d05-695b586a1fac\") "
Nov 02 20:08:24.995725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.995673    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca" pod "7d4eda95-3565-4af4-8d05-695b586a1fac" (UID: "7d4eda95-3565-4af4-8d05-695b586a1fac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:24.995864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.995847    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:24.997619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.997594    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca" pod "7d4eda95-3565-4af4-8d05-695b586a1fac" (UID: "7d4eda95-3565-4af4-8d05-695b586a1fac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:24.997795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:24.997740    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr" pod "7d4eda95-3565-4af4-8d05-695b586a1fac" (UID: "7d4eda95-3565-4af4-8d05-695b586a1fac"). InnerVolumeSpecName "kube-api-access-zc4rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:25.096489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.096453    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4rr\" (UniqueName: \"kubernetes.io/projected/7d4eda95-3565-4af4-8d05-695b586a1fac-kube-api-access-zc4rr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:25.096489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.096485    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4eda95-3565-4af4-8d05-695b586a1fac-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:25.678286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.678240    5099 generic.go:356] "Generic (PLEG): container finished" podID="7cade08b-806e-4545-99a8-188f8c78d0ba" containerID="a31ad549decaf8b467881ef9a0c9b5873006c404923692a669dac8c86444104b" exitCode=0
Nov 02 20:08:25.678772 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.678332    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v" event={"ID":"7cade08b-806e-4545-99a8-188f8c78d0ba","Type":"ContainerDied","Data":"a31ad549decaf8b467881ef9a0c9b5873006c404923692a669dac8c86444104b"}
Nov 02 20:08:25.679928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.679909    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"
Nov 02 20:08:25.680037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.679959    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq" event={"ID":"7d4eda95-3565-4af4-8d05-695b586a1fac","Type":"ContainerDied","Data":"066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc"}
Nov 02 20:08:25.680037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:25.680009    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066ee03e6432337d8afddbd7565cb2e765d7c245486512a9bd04071336dd0ccc"
Nov 02 20:08:26.841269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.841242    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:26.913103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.913069    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\") pod \"7cade08b-806e-4545-99a8-188f8c78d0ba\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") "
Nov 02 20:08:26.913268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.913116    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\" (UniqueName: \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\") pod \"7cade08b-806e-4545-99a8-188f8c78d0ba\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") "
Nov 02 20:08:26.913268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.913180    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\") pod \"7cade08b-806e-4545-99a8-188f8c78d0ba\" (UID: \"7cade08b-806e-4545-99a8-188f8c78d0ba\") "
Nov 02 20:08:26.913615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.913587    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca" pod "7cade08b-806e-4545-99a8-188f8c78d0ba" (UID: "7cade08b-806e-4545-99a8-188f8c78d0ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:26.915216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.915189    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca" pod "7cade08b-806e-4545-99a8-188f8c78d0ba" (UID: "7cade08b-806e-4545-99a8-188f8c78d0ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:26.915302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:26.915279    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q" pod "7cade08b-806e-4545-99a8-188f8c78d0ba" (UID: "7cade08b-806e-4545-99a8-188f8c78d0ba"). InnerVolumeSpecName "kube-api-access-vht7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:27.037095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.037009    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:27.038750 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.038508    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vht7q\" (UniqueName: \"kubernetes.io/projected/7cade08b-806e-4545-99a8-188f8c78d0ba-kube-api-access-vht7q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:27.038750 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.038550    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cade08b-806e-4545-99a8-188f8c78d0ba-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:27.692854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.692823    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"
Nov 02 20:08:27.693056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.692850    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v" event={"ID":"7cade08b-806e-4545-99a8-188f8c78d0ba","Type":"ContainerDied","Data":"a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828"}
Nov 02 20:08:27.693056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:27.692901    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9241f6c916294b49059533df6e593c61e6624c37a3f6a41c274faf1e896c828"
Nov 02 20:08:31.300795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:31.300750    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 20:08:31.301360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:31.300992    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy" containerID="cri-o://2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff" gracePeriod=30
Nov 02 20:08:35.764820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.764764    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"]
Nov 02 20:08:35.765374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765357    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7d4eda95-3565-4af4-8d05-695b586a1fac" containerName="revoke-keys"
Nov 02 20:08:35.765425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765375    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4eda95-3565-4af4-8d05-695b586a1fac" containerName="revoke-keys"
Nov 02 20:08:35.765425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765393    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7cade08b-806e-4545-99a8-188f8c78d0ba" containerName="revoke-keys"
Nov 02 20:08:35.765425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765399    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cade08b-806e-4545-99a8-188f8c78d0ba" containerName="revoke-keys"
Nov 02 20:08:35.765516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765507    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4eda95-3565-4af4-8d05-695b586a1fac" containerName="revoke-keys"
Nov 02 20:08:35.765554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.765517    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cade08b-806e-4545-99a8-188f8c78d0ba" containerName="revoke-keys"
Nov 02 20:08:35.768807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.768774    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.821238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.821209    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"]
Nov 02 20:08:35.834714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.833470    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.834714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.833613    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqll\" (UniqueName: \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.834714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.833702    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.865255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.865225    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:35.865395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.865278    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:35.934633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.934601    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.934873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.934661    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqll\" (UniqueName: \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.934873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.934743    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.935519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.935493    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:35.936961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:35.936943    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:36.001516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.001429    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqll\" (UniqueName: \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\") pod \"maas-api-revoke-sub-c414e206-ssl7t\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:36.081929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.081839    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:36.446003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.445961    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"]
Nov 02 20:08:36.449650 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:36.449606    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1be8ff_bfbb_40e6_8c2d_2b3cf26096ce.slice/crio-0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0 WatchSource:0}: Error finding container 0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0: Status 404 returned error can't find the container with id 0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0
Nov 02 20:08:36.554097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.554074    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 20:08:36.643884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.643853    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.643884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.643887    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644012    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644041    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644071    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644104    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644155    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644143    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644192    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644234    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644269    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\") pod \"470b442b-bfe5-48dd-97e2-1b3581f33f28\" (UID: \"470b442b-bfe5-48dd-97e2-1b3581f33f28\") "
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644335    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:36.644371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644351    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:36.644759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644737    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.644849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644764    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.644849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644793    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.644849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644807    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.646919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.644891    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:36.646919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.646631    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:08:36.646919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.646735    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:08:36.646919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.646828    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:36.647177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.646977    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58" pod "470b442b-bfe5-48dd-97e2-1b3581f33f28" (UID: "470b442b-bfe5-48dd-97e2-1b3581f33f28"). InnerVolumeSpecName "kube-api-access-d5g58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:36.745626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.745547    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.745626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.745576    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5g58\" (UniqueName: \"kubernetes.io/projected/470b442b-bfe5-48dd-97e2-1b3581f33f28-kube-api-access-d5g58\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.745626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.745589    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.745626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.745602    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b442b-bfe5-48dd-97e2-1b3581f33f28-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.745626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.745615    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b442b-bfe5-48dd-97e2-1b3581f33f28-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:36.748384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.748346    5099 generic.go:356] "Generic (PLEG): container finished" podID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerID="2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff" exitCode=0
Nov 02 20:08:36.748538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.748401    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"
Nov 02 20:08:36.748538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.748428    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerDied","Data":"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"}
Nov 02 20:08:36.748538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.748494    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn" event={"ID":"470b442b-bfe5-48dd-97e2-1b3581f33f28","Type":"ContainerDied","Data":"e165c402ca731abeed48294cfa5681c60b0fcad0d81b24e4bd7f1dcc20dc7b2d"}
Nov 02 20:08:36.748538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.748522    5099 scope.go:122] "RemoveContainer" containerID="2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"
Nov 02 20:08:36.750061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.750039    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t" event={"ID":"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce","Type":"ContainerStarted","Data":"0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0"}
Nov 02 20:08:36.758189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.758164    5099 scope.go:122] "RemoveContainer" containerID="02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"
Nov 02 20:08:36.767066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.767048    5099 scope.go:122] "RemoveContainer" containerID="2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"
Nov 02 20:08:36.767359 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:08:36.767316    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff\": container with ID starting with 2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff not found: ID does not exist" containerID="2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"
Nov 02 20:08:36.767359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.767340    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff"} err="failed to get container status \"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff\": rpc error: code = NotFound desc = could not find container \"2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff\": container with ID starting with 2aef5b1cb4be19d8bb1f2a154b453640cbb8c6f6aa50fc79713408eeb73666ff not found: ID does not exist"
Nov 02 20:08:36.767359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.767358    5099 scope.go:122] "RemoveContainer" containerID="02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"
Nov 02 20:08:36.767605 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:08:36.767584    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036\": container with ID starting with 02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036 not found: ID does not exist" containerID="02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"
Nov 02 20:08:36.767655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.767615    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036"} err="failed to get container status \"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036\": rpc error: code = NotFound desc = could not find container \"02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036\": container with ID starting with 02c96b014604729a514ba6d3df68908abb5ebe5d6cedfd538b6cac7e5ef08036 not found: ID does not exist"
Nov 02 20:08:36.905611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.905460    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 20:08:36.908919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:36.908891    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jbvgn"]
Nov 02 20:08:37.258578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:37.258546    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" path="/var/lib/kubelet/pods/470b442b-bfe5-48dd-97e2-1b3581f33f28/volumes"
Nov 02 20:08:37.758591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:37.758557    5099 generic.go:356] "Generic (PLEG): container finished" podID="3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" containerID="e4a95cba23899db03e9624a7c630d59560f78ed008f29492741fbcc649147e67" exitCode=0
Nov 02 20:08:37.758773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:37.758645    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t" event={"ID":"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce","Type":"ContainerDied","Data":"e4a95cba23899db03e9624a7c630d59560f78ed008f29492741fbcc649147e67"}
Nov 02 20:08:38.912107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.912079    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:38.969725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.969684    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\" (UniqueName: \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\") pod \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") "
Nov 02 20:08:38.969994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.969739    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\") pod \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") "
Nov 02 20:08:38.969994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.969940    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\") pod \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\" (UID: \"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce\") "
Nov 02 20:08:38.970159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.970135    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca" pod "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" (UID: "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:38.970386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.970366    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:38.971964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.971936    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca" pod "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" (UID: "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:38.972095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:38.972074    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll" pod "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" (UID: "3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce"). InnerVolumeSpecName "kube-api-access-xgqll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:39.071340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:39.071253    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:39.071340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:39.071288    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqll\" (UniqueName: \"kubernetes.io/projected/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce-kube-api-access-xgqll\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:39.772645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:39.772617    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"
Nov 02 20:08:39.772645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:39.772623    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t" event={"ID":"3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce","Type":"ContainerDied","Data":"0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0"}
Nov 02 20:08:39.772921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:39.772675    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b00a80ac0a76609d39947b6cff73976093ba801465b45a456b5ba5dee6c62e0"
Nov 02 20:08:41.056448 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.056416    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"]
Nov 02 20:08:41.056976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.056960    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" containerName="revoke-keys"
Nov 02 20:08:41.057030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.056977    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" containerName="revoke-keys"
Nov 02 20:08:41.057030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.057001    5099 cpu_manager.go:414] "containerMap: removing container" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:41.057030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.057006    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:41.057129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.057091    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" containerName="revoke-keys"
Nov 02 20:08:41.057129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.057104    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:41.057129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.057111    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:41.059331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.059313    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.093964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.093922    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnh7\" (UniqueName: \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.094127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.094103    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.094224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.094207    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.113443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.113410    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"]
Nov 02 20:08:41.156313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.156272    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:41.156461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.156337    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:41.195314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.195281    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnh7\" (UniqueName: \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.195516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.195380    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.195516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.195425    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.196275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.196247    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.198698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.198673    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.251533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.251507    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnh7\" (UniqueName: \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\") pod \"maas-api-revoke-sub-fc9a1063-jgt8m\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.371931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.371898    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:41.783137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:41.783052    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"]
Nov 02 20:08:41.789411 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:41.789377    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf30ab4f_ad0d_4381_840a_1130d2947d59.slice/crio-30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241 WatchSource:0}: Error finding container 30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241: Status 404 returned error can't find the container with id 30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241
Nov 02 20:08:42.799211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:42.799177    5099 generic.go:356] "Generic (PLEG): container finished" podID="cf30ab4f-ad0d-4381-840a-1130d2947d59" containerID="b71338c07104f1d6279e7432f763a4c25814d43a6c9e0bd0ce32ffac03e4ad3c" exitCode=0
Nov 02 20:08:42.799618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:42.799260    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m" event={"ID":"cf30ab4f-ad0d-4381-840a-1130d2947d59","Type":"ContainerDied","Data":"b71338c07104f1d6279e7432f763a4c25814d43a6c9e0bd0ce32ffac03e4ad3c"}
Nov 02 20:08:42.799618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:42.799324    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m" event={"ID":"cf30ab4f-ad0d-4381-840a-1130d2947d59","Type":"ContainerStarted","Data":"30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241"}
Nov 02 20:08:43.953630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:43.953600    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:44.026923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.026890    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\") pod \"cf30ab4f-ad0d-4381-840a-1130d2947d59\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") "
Nov 02 20:08:44.027065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.026957    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\" (UniqueName: \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\") pod \"cf30ab4f-ad0d-4381-840a-1130d2947d59\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") "
Nov 02 20:08:44.027065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.027023    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\") pod \"cf30ab4f-ad0d-4381-840a-1130d2947d59\" (UID: \"cf30ab4f-ad0d-4381-840a-1130d2947d59\") "
Nov 02 20:08:44.027402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.027366    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca" pod "cf30ab4f-ad0d-4381-840a-1130d2947d59" (UID: "cf30ab4f-ad0d-4381-840a-1130d2947d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:44.029103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.029076    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7" pod "cf30ab4f-ad0d-4381-840a-1130d2947d59" (UID: "cf30ab4f-ad0d-4381-840a-1130d2947d59"). InnerVolumeSpecName "kube-api-access-klnh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:44.029103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.029086    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca" pod "cf30ab4f-ad0d-4381-840a-1130d2947d59" (UID: "cf30ab4f-ad0d-4381-840a-1130d2947d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:44.128693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.128661    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:44.128693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.128691    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnh7\" (UniqueName: \"kubernetes.io/projected/cf30ab4f-ad0d-4381-840a-1130d2947d59-kube-api-access-klnh7\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:44.128913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.128701    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf30ab4f-ad0d-4381-840a-1130d2947d59-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:44.813042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.813011    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"
Nov 02 20:08:44.813194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.813010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m" event={"ID":"cf30ab4f-ad0d-4381-840a-1130d2947d59","Type":"ContainerDied","Data":"30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241"}
Nov 02 20:08:44.813194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:44.813134    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30495156839aa1b85d9bf027ae56cd5fd6257afa235a2258351ed01bfe140241"
Nov 02 20:08:46.096078 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:08:46.096042    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:08:46.500755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.500712    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"]
Nov 02 20:08:46.501362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.501340    5099 cpu_manager.go:414] "containerMap: removing container" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:46.501362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.501361    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470b442b-bfe5-48dd-97e2-1b3581f33f28" containerName="istio-proxy"
Nov 02 20:08:46.501513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.501390    5099 cpu_manager.go:414] "containerMap: removing container" podUID="cf30ab4f-ad0d-4381-840a-1130d2947d59" containerName="revoke-keys"
Nov 02 20:08:46.501513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.501397    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf30ab4f-ad0d-4381-840a-1130d2947d59" containerName="revoke-keys"
Nov 02 20:08:46.501608 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.501546    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf30ab4f-ad0d-4381-840a-1130d2947d59" containerName="revoke-keys"
Nov 02 20:08:46.503743 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.503722    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.554240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.554193    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.554407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.554241    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqcp\" (UniqueName: \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.554481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.554397    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.600474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.600443    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:46.600474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.600466    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:46.605251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.605225    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"]
Nov 02 20:08:46.608196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.608174    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"]
Nov 02 20:08:46.611721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.611703    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.655391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655360    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.655543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655513    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.655593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655543    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqcp\" (UniqueName: \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.655631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655589    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9dr\" (UniqueName: \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.655631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655623    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.655742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.655654    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.656291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.656263    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.658530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.658506    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.662771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.662736    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"]
Nov 02 20:08:46.709293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.709265    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqcp\" (UniqueName: \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\") pod \"maas-api-revoke-sub-76fc7448-vjljs\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.756969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.756884    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9dr\" (UniqueName: \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.756969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.756956    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.757180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.757035    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.757757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.757731    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.759716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.759693    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.810315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.810287    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9dr\" (UniqueName: \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\") pod \"maas-api-revoke-sub-bf439863-n46dc\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:46.816129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.816107    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:46.924246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:46.924219    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:47.187185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.187153    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"]
Nov 02 20:08:47.188420 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:47.188388    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c91fc7_1fcd_440b_8e75_535a3991cedb.slice/crio-263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2 WatchSource:0}: Error finding container 263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2: Status 404 returned error can't find the container with id 263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2
Nov 02 20:08:47.293892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.293867    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"]
Nov 02 20:08:47.294420 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:47.294398    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17becb9_5fdc_481f_9fd2_c0fb61a12722.slice/crio-d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf WatchSource:0}: Error finding container d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf: Status 404 returned error can't find the container with id d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf
Nov 02 20:08:47.834840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.834740    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs" event={"ID":"d5c91fc7-1fcd-440b-8e75-535a3991cedb","Type":"ContainerStarted","Data":"aaca9ad48d485f990d1a3647ed25e922e12d4a58d10e2be736584405763dd7f6"}
Nov 02 20:08:47.834840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.834816    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs" event={"ID":"d5c91fc7-1fcd-440b-8e75-535a3991cedb","Type":"ContainerStarted","Data":"263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2"}
Nov 02 20:08:47.836329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.836304    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc" event={"ID":"e17becb9-5fdc-481f-9fd2-c0fb61a12722","Type":"ContainerStarted","Data":"42ccbbac221925c1379a98035f8fa951d539a0a8fba0ae4f4e4083189946cdb0"}
Nov 02 20:08:47.836433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.836344    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc" event={"ID":"e17becb9-5fdc-481f-9fd2-c0fb61a12722","Type":"ContainerStarted","Data":"d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf"}
Nov 02 20:08:47.944864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:47.944669    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs" podStartSLOduration=1.944648819 podStartE2EDuration="1.944648819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:47.941407228 +0000 UTC m=+1016.815678117" watchObservedRunningTime="2026-11-02 20:08:47.944648819 +0000 UTC m=+1016.818919707"
Nov 02 20:08:48.048297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.048236    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc" podStartSLOduration=2.048218297 podStartE2EDuration="2.048218297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:48.046598639 +0000 UTC m=+1016.920869528" watchObservedRunningTime="2026-11-02 20:08:48.048218297 +0000 UTC m=+1016.922489185"
Nov 02 20:08:48.761968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.761932    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"]
Nov 02 20:08:48.765087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.765067    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.843676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.843634    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"]
Nov 02 20:08:48.845634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.845607    5099 generic.go:356] "Generic (PLEG): container finished" podID="d5c91fc7-1fcd-440b-8e75-535a3991cedb" containerID="aaca9ad48d485f990d1a3647ed25e922e12d4a58d10e2be736584405763dd7f6" exitCode=0
Nov 02 20:08:48.845798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.845688    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs" event={"ID":"d5c91fc7-1fcd-440b-8e75-535a3991cedb","Type":"ContainerDied","Data":"aaca9ad48d485f990d1a3647ed25e922e12d4a58d10e2be736584405763dd7f6"}
Nov 02 20:08:48.848086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.848060    5099 generic.go:356] "Generic (PLEG): container finished" podID="e17becb9-5fdc-481f-9fd2-c0fb61a12722" containerID="42ccbbac221925c1379a98035f8fa951d539a0a8fba0ae4f4e4083189946cdb0" exitCode=0
Nov 02 20:08:48.848189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.848105    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc" event={"ID":"e17becb9-5fdc-481f-9fd2-c0fb61a12722","Type":"ContainerDied","Data":"42ccbbac221925c1379a98035f8fa951d539a0a8fba0ae4f4e4083189946cdb0"}
Nov 02 20:08:48.886512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.886486    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.886639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.886528    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgjz\" (UniqueName: \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.886639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.886596    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.988289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.988257    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgjz\" (UniqueName: \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.988289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.988293    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.988494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.988405    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.989019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.988992    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:48.990737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:48.990712    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:49.043557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.043492    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgjz\" (UniqueName: \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\") pod \"maas-api-revoke-sub-dab0e09a-5jr87\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:49.079727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.079699    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:49.447611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.447587    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"]
Nov 02 20:08:49.448192 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:49.448161    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1567b2e_c7e2_43ad_ad61_f183144fdacd.slice/crio-473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e WatchSource:0}: Error finding container 473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e: Status 404 returned error can't find the container with id 473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e
Nov 02 20:08:49.856164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.856127    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87" event={"ID":"d1567b2e-c7e2-43ad-ad61-f183144fdacd","Type":"ContainerStarted","Data":"9b31b67e414c059792710005c5f35e17a459ee12311c7f10c0c725f5e965be06"}
Nov 02 20:08:49.856598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.856188    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87" event={"ID":"d1567b2e-c7e2-43ad-ad61-f183144fdacd","Type":"ContainerStarted","Data":"473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e"}
Nov 02 20:08:49.966730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:49.966683    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87" podStartSLOduration=1.9666694580000001 podStartE2EDuration="1.966669458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:49.963965985 +0000 UTC m=+1018.838236875" watchObservedRunningTime="2026-11-02 20:08:49.966669458 +0000 UTC m=+1018.840940343"
Nov 02 20:08:50.011665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.011633    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:50.046038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.046015    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:50.101715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.101689    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\") pod \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") "
Nov 02 20:08:50.101912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.101736    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\" (UniqueName: \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\") pod \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") "
Nov 02 20:08:50.101912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.101778    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\") pod \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") "
Nov 02 20:08:50.101912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.101849    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\" (UniqueName: \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\") pod \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\" (UID: \"d5c91fc7-1fcd-440b-8e75-535a3991cedb\") "
Nov 02 20:08:50.102076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.101977    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\") pod \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") "
Nov 02 20:08:50.102076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.102064    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\") pod \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\" (UID: \"e17becb9-5fdc-481f-9fd2-c0fb61a12722\") "
Nov 02 20:08:50.102227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.102187    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca" pod "d5c91fc7-1fcd-440b-8e75-535a3991cedb" (UID: "d5c91fc7-1fcd-440b-8e75-535a3991cedb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:50.102490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.102456    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca" pod "e17becb9-5fdc-481f-9fd2-c0fb61a12722" (UID: "e17becb9-5fdc-481f-9fd2-c0fb61a12722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:50.102616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.102522    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.104061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.104040    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca" pod "d5c91fc7-1fcd-440b-8e75-535a3991cedb" (UID: "d5c91fc7-1fcd-440b-8e75-535a3991cedb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:50.104161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.104089    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp" pod "d5c91fc7-1fcd-440b-8e75-535a3991cedb" (UID: "d5c91fc7-1fcd-440b-8e75-535a3991cedb"). InnerVolumeSpecName "kube-api-access-lzqcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:50.104308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.104289    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca" pod "e17becb9-5fdc-481f-9fd2-c0fb61a12722" (UID: "e17becb9-5fdc-481f-9fd2-c0fb61a12722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:50.104351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.104314    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr" pod "e17becb9-5fdc-481f-9fd2-c0fb61a12722" (UID: "e17becb9-5fdc-481f-9fd2-c0fb61a12722"). InnerVolumeSpecName "kube-api-access-tb9dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:50.203825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.203772    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c91fc7-1fcd-440b-8e75-535a3991cedb-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.203825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.203825    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9dr\" (UniqueName: \"kubernetes.io/projected/e17becb9-5fdc-481f-9fd2-c0fb61a12722-kube-api-access-tb9dr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.204020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.203836    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqcp\" (UniqueName: \"kubernetes.io/projected/d5c91fc7-1fcd-440b-8e75-535a3991cedb-kube-api-access-lzqcp\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.204020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.203845    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.204020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.203854    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17becb9-5fdc-481f-9fd2-c0fb61a12722-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:50.863889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.863848    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"
Nov 02 20:08:50.863889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.863867    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc" event={"ID":"e17becb9-5fdc-481f-9fd2-c0fb61a12722","Type":"ContainerDied","Data":"d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf"}
Nov 02 20:08:50.864361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.863928    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d149312b029782ee6b5d20c3e744df4d4f9daafa6825b99ad0b343035abecf"
Nov 02 20:08:50.866544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.866517    5099 generic.go:356] "Generic (PLEG): container finished" podID="d1567b2e-c7e2-43ad-ad61-f183144fdacd" containerID="9b31b67e414c059792710005c5f35e17a459ee12311c7f10c0c725f5e965be06" exitCode=0
Nov 02 20:08:50.866657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.866603    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87" event={"ID":"d1567b2e-c7e2-43ad-ad61-f183144fdacd","Type":"ContainerDied","Data":"9b31b67e414c059792710005c5f35e17a459ee12311c7f10c0c725f5e965be06"}
Nov 02 20:08:50.868061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.868041    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs" event={"ID":"d5c91fc7-1fcd-440b-8e75-535a3991cedb","Type":"ContainerDied","Data":"263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2"}
Nov 02 20:08:50.868127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.868072    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"
Nov 02 20:08:50.868183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:50.868076    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263dbc3832a2ea60e656cc00362d38e90928110bf49c1ca23506ec2c8f6d7da2"
Nov 02 20:08:52.029104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.029082    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:52.123418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.123385    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\" (UniqueName: \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\") pod \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") "
Nov 02 20:08:52.123583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.123437    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\") pod \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") "
Nov 02 20:08:52.123621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.123590    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\") pod \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\" (UID: \"d1567b2e-c7e2-43ad-ad61-f183144fdacd\") "
Nov 02 20:08:52.123985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.123956    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca" pod "d1567b2e-c7e2-43ad-ad61-f183144fdacd" (UID: "d1567b2e-c7e2-43ad-ad61-f183144fdacd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:08:52.125523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.125476    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca" pod "d1567b2e-c7e2-43ad-ad61-f183144fdacd" (UID: "d1567b2e-c7e2-43ad-ad61-f183144fdacd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:08:52.125597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.125568    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz" pod "d1567b2e-c7e2-43ad-ad61-f183144fdacd" (UID: "d1567b2e-c7e2-43ad-ad61-f183144fdacd"). InnerVolumeSpecName "kube-api-access-7sgjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:08:52.224561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.224530    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:52.224561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.224558    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sgjz\" (UniqueName: \"kubernetes.io/projected/d1567b2e-c7e2-43ad-ad61-f183144fdacd-kube-api-access-7sgjz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:52.224770 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.224590    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1567b2e-c7e2-43ad-ad61-f183144fdacd-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:08:52.881659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.881632    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"
Nov 02 20:08:52.881849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.881670    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87" event={"ID":"d1567b2e-c7e2-43ad-ad61-f183144fdacd","Type":"ContainerDied","Data":"473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e"}
Nov 02 20:08:52.881849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:52.881726    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473072243c31c0a8dba0fb0d417b2b9d182f63ef3630b93be3a99e8e78d1552e"
Nov 02 20:08:57.458798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.458749    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"]
Nov 02 20:08:57.459317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459301    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d5c91fc7-1fcd-440b-8e75-535a3991cedb" containerName="revoke-keys"
Nov 02 20:08:57.459360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459318    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c91fc7-1fcd-440b-8e75-535a3991cedb" containerName="revoke-keys"
Nov 02 20:08:57.459360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459337    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e17becb9-5fdc-481f-9fd2-c0fb61a12722" containerName="revoke-keys"
Nov 02 20:08:57.459360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459343    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17becb9-5fdc-481f-9fd2-c0fb61a12722" containerName="revoke-keys"
Nov 02 20:08:57.459456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459360    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d1567b2e-c7e2-43ad-ad61-f183144fdacd" containerName="revoke-keys"
Nov 02 20:08:57.459456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459366    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1567b2e-c7e2-43ad-ad61-f183144fdacd" containerName="revoke-keys"
Nov 02 20:08:57.459520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459465    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c91fc7-1fcd-440b-8e75-535a3991cedb" containerName="revoke-keys"
Nov 02 20:08:57.459520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459477    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17becb9-5fdc-481f-9fd2-c0fb61a12722" containerName="revoke-keys"
Nov 02 20:08:57.459520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.459485    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1567b2e-c7e2-43ad-ad61-f183144fdacd" containerName="revoke-keys"
Nov 02 20:08:57.461962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.461944    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.516371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.516342    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"]
Nov 02 20:08:57.559179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.559144    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:08:57.559320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.559253    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:08:57.582381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.582347    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.582561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.582410    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltj5b\" (UniqueName: \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.582561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.582521    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.683390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.683356    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltj5b\" (UniqueName: \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.683573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.683415    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.683573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.683490    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.684141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.684115    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.685914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.685893    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.736861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.736771    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltj5b\" (UniqueName: \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\") pod \"maas-api-revoke-sub-5cb53246-j6f9s\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:57.774407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:57.774377    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:08:58.139453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:58.139423    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"]
Nov 02 20:08:58.140247 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:08:58.140207    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7569deda_479a_4eef_8175_b82491820747.slice/crio-79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809 WatchSource:0}: Error finding container 79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809: Status 404 returned error can't find the container with id 79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809
Nov 02 20:08:58.920388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:58.920346    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s" event={"ID":"7569deda-479a-4eef-8175-b82491820747","Type":"ContainerStarted","Data":"efa1e71987533e29150d38f615e66aba1eec505a44fff2e8ee8d7309a49d751d"}
Nov 02 20:08:58.920777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:58.920405    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s" event={"ID":"7569deda-479a-4eef-8175-b82491820747","Type":"ContainerStarted","Data":"79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809"}
Nov 02 20:08:59.029900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:59.029839    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s" podStartSLOduration=2.029822078 podStartE2EDuration="2.029822078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:59.027137617 +0000 UTC m=+1027.901408506" watchObservedRunningTime="2026-11-02 20:08:59.029822078 +0000 UTC m=+1027.904092971"
Nov 02 20:08:59.927718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:59.927634    5099 generic.go:356] "Generic (PLEG): container finished" podID="7569deda-479a-4eef-8175-b82491820747" containerID="efa1e71987533e29150d38f615e66aba1eec505a44fff2e8ee8d7309a49d751d" exitCode=0
Nov 02 20:08:59.927718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:08:59.927663    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s" event={"ID":"7569deda-479a-4eef-8175-b82491820747","Type":"ContainerDied","Data":"efa1e71987533e29150d38f615e66aba1eec505a44fff2e8ee8d7309a49d751d"}
Nov 02 20:09:01.085991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.085960    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:09:01.222571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.222482    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\" (UniqueName: \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\") pod \"7569deda-479a-4eef-8175-b82491820747\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") "
Nov 02 20:09:01.222737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.222693    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\") pod \"7569deda-479a-4eef-8175-b82491820747\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") "
Nov 02 20:09:01.222827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.222754    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\") pod \"7569deda-479a-4eef-8175-b82491820747\" (UID: \"7569deda-479a-4eef-8175-b82491820747\") "
Nov 02 20:09:01.223135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.223106    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca" pod "7569deda-479a-4eef-8175-b82491820747" (UID: "7569deda-479a-4eef-8175-b82491820747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:01.223286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.223271    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7569deda-479a-4eef-8175-b82491820747-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:01.225313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.225279    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b" pod "7569deda-479a-4eef-8175-b82491820747" (UID: "7569deda-479a-4eef-8175-b82491820747"). InnerVolumeSpecName "kube-api-access-ltj5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:01.225445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.225394    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca" pod "7569deda-479a-4eef-8175-b82491820747" (UID: "7569deda-479a-4eef-8175-b82491820747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:01.323963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.323928    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltj5b\" (UniqueName: \"kubernetes.io/projected/7569deda-479a-4eef-8175-b82491820747-kube-api-access-ltj5b\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:01.323963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.323957    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7569deda-479a-4eef-8175-b82491820747-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:01.942488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.942451    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"
Nov 02 20:09:01.942488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.942463    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s" event={"ID":"7569deda-479a-4eef-8175-b82491820747","Type":"ContainerDied","Data":"79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809"}
Nov 02 20:09:01.942683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:01.942518    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79240c8f7c11c0d3dc08544104702ba7c16d8e5b437b8444ef53040fb53d8809"
Nov 02 20:09:02.146400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.146363    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:09:02.146943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.146927    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7569deda-479a-4eef-8175-b82491820747" containerName="revoke-keys"
Nov 02 20:09:02.146997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.146945    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7569deda-479a-4eef-8175-b82491820747" containerName="revoke-keys"
Nov 02 20:09:02.147075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.147066    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7569deda-479a-4eef-8175-b82491820747" containerName="revoke-keys"
Nov 02 20:09:02.149498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.149480    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.207243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.207162    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:09:02.249180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.248562    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"distinct-w4-cc64d3-kserve-self-signed-certs\""
Nov 02 20:09:02.249180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.248576    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"openshift-service-ca.crt\""
Nov 02 20:09:02.249180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.248594    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"kube-root-ca.crt\""
Nov 02 20:09:02.249180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.248636    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"default-dockercfg-j9ncr\""
Nov 02 20:09:02.335152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335117    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.335332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335180    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.335390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335337    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.335430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335402    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.335515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335498    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.335664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.335648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crx9v\" (UniqueName: \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436382    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436483    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crx9v\" (UniqueName: \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436527    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436599    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436649    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.436713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.436699    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.437488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.437017    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.437488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.437060    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.437488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.437139    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.438777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.438751    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.439045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.439028    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.490910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.490838    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crx9v\" (UniqueName: \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\") pod \"distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:02.762119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:02.762024    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:03.001335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.001294    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"]
Nov 02 20:09:03.005352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.005329    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.061568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.061534    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"]
Nov 02 20:09:03.100499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.100460    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:03.100637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.100575    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:09:03.148657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.148628    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtwm\" (UniqueName: \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.149088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.148692    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.149088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.148868    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.160188 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:03.160154    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0257175b_5e44_485d_a684_dc8024f7a6d8.slice/crio-8038b73910f2c7e9cb984f3ce13479ed4eac45dacc3640a5b6ac3cd904381f1e WatchSource:0}: Error finding container 8038b73910f2c7e9cb984f3ce13479ed4eac45dacc3640a5b6ac3cd904381f1e: Status 404 returned error can't find the container with id 8038b73910f2c7e9cb984f3ce13479ed4eac45dacc3640a5b6ac3cd904381f1e
Nov 02 20:09:03.160494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.160470    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:09:03.249426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.249396    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.249579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.249562    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.249628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.249615    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtwm\" (UniqueName: \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.250148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.250124    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.251733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.251706    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.303182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.303114    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtwm\" (UniqueName: \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\") pod \"maas-api-revoke-sub-3d2c1d43-2ssjv\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.319907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.319885    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:03.685364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.685334    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"]
Nov 02 20:09:03.686947 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:03.686920    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f00c3c_682c_406d_a845_44dc4baaa920.slice/crio-6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b WatchSource:0}: Error finding container 6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b: Status 404 returned error can't find the container with id 6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b
Nov 02 20:09:03.946606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.946536    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"]
Nov 02 20:09:03.950109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.950082    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:03.959863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.959728    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" event={"ID":"0257175b-5e44-485d-a684-dc8024f7a6d8","Type":"ContainerStarted","Data":"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"}
Nov 02 20:09:03.959863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.959770    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" event={"ID":"0257175b-5e44-485d-a684-dc8024f7a6d8","Type":"ContainerStarted","Data":"8038b73910f2c7e9cb984f3ce13479ed4eac45dacc3640a5b6ac3cd904381f1e"}
Nov 02 20:09:03.962283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.962221    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv" event={"ID":"f6f00c3c-682c-406d-a845-44dc4baaa920","Type":"ContainerStarted","Data":"6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b"}
Nov 02 20:09:03.969700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.969677    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:03.969823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.969709    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4sf\" (UniqueName: \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:03.969823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:03.969730    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.008389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.008353    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"]
Nov 02 20:09:04.090990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.090776    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.090990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.090845    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4sf\" (UniqueName: \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.090990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.090884    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.098180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.094251    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.098180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.095014    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.150816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.149945    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4sf\" (UniqueName: \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\") pod \"maas-api-revoke-sub-0d785604-9wmqv\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.176117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.176049    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podStartSLOduration=2.176029694 podStartE2EDuration="2.176029694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:04.172939511 +0000 UTC m=+1033.047210400" watchObservedRunningTime="2026-11-02 20:09:04.176029694 +0000 UTC m=+1033.050300584"
Nov 02 20:09:04.267082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.266991    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:04.680530 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:04.679543    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d3e622_6474_46a8_a673_5e0225c9a7d4.slice/crio-ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80 WatchSource:0}: Error finding container ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80: Status 404 returned error can't find the container with id ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80
Nov 02 20:09:04.680530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.680452    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"]
Nov 02 20:09:04.762224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.762195    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:04.969703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.969671    5099 generic.go:356] "Generic (PLEG): container finished" podID="f6f00c3c-682c-406d-a845-44dc4baaa920" containerID="3c2f20c637ea93fe50308be08e648f7a256c976dc077e437eefbb9e3aa650fcb" exitCode=0
Nov 02 20:09:04.969869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.969751    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv" event={"ID":"f6f00c3c-682c-406d-a845-44dc4baaa920","Type":"ContainerDied","Data":"3c2f20c637ea93fe50308be08e648f7a256c976dc077e437eefbb9e3aa650fcb"}
Nov 02 20:09:04.971406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.971381    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv" event={"ID":"95d3e622-6474-46a8-a673-5e0225c9a7d4","Type":"ContainerStarted","Data":"dd563e9839d6e9c1c53b64c0a78d6df5067d82b33a047429787cac41f345ee2d"}
Nov 02 20:09:04.971508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:04.971427    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv" event={"ID":"95d3e622-6474-46a8-a673-5e0225c9a7d4","Type":"ContainerStarted","Data":"ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80"}
Nov 02 20:09:05.183011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.182959    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv" podStartSLOduration=2.18294063 podStartE2EDuration="2.18294063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:05.182647254 +0000 UTC m=+1034.056918144" watchObservedRunningTime="2026-11-02 20:09:05.18294063 +0000 UTC m=+1034.057211520"
Nov 02 20:09:05.940631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.940600    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lcjhp-t56zv"]
Nov 02 20:09:05.943510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.943478    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:05.979897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.979862    5099 generic.go:356] "Generic (PLEG): container finished" podID="95d3e622-6474-46a8-a673-5e0225c9a7d4" containerID="dd563e9839d6e9c1c53b64c0a78d6df5067d82b33a047429787cac41f345ee2d" exitCode=0
Nov 02 20:09:05.980054 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.979949    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv" event={"ID":"95d3e622-6474-46a8-a673-5e0225c9a7d4","Type":"ContainerDied","Data":"dd563e9839d6e9c1c53b64c0a78d6df5067d82b33a047429787cac41f345ee2d"}
Nov 02 20:09:05.996123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:05.996091    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lcjhp-t56zv"]
Nov 02 20:09:06.023817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.023769    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.023996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.023923    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdht\" (UniqueName: \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.024082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.024002    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.024998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.024973    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.126494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.126457    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.126679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.126560    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.126679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.126637    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdht\" (UniqueName: \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.126679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.126668    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.126915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.126887    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.127236 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.127216    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.127539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.127513    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.152937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.152917    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:06.180518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.180480    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdht\" (UniqueName: \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\") pod \"periodic-gathering-lcjhp-t56zv\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") " pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.228022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.227946    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\" (UniqueName: \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\") pod \"f6f00c3c-682c-406d-a845-44dc4baaa920\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") "
Nov 02 20:09:06.228416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.228084    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\") pod \"f6f00c3c-682c-406d-a845-44dc4baaa920\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") "
Nov 02 20:09:06.228416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.228191    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\") pod \"f6f00c3c-682c-406d-a845-44dc4baaa920\" (UID: \"f6f00c3c-682c-406d-a845-44dc4baaa920\") "
Nov 02 20:09:06.228596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.228575    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca" pod "f6f00c3c-682c-406d-a845-44dc4baaa920" (UID: "f6f00c3c-682c-406d-a845-44dc4baaa920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:06.230189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.230166    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm" pod "f6f00c3c-682c-406d-a845-44dc4baaa920" (UID: "f6f00c3c-682c-406d-a845-44dc4baaa920"). InnerVolumeSpecName "kube-api-access-rxtwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:06.230286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.230211    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca" pod "f6f00c3c-682c-406d-a845-44dc4baaa920" (UID: "f6f00c3c-682c-406d-a845-44dc4baaa920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:06.254467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.254437    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:09:06.329767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.329732    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:06.329916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.329771    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f00c3c-682c-406d-a845-44dc4baaa920-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:06.329916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.329807    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtwm\" (UniqueName: \"kubernetes.io/projected/f6f00c3c-682c-406d-a845-44dc4baaa920-kube-api-access-rxtwm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:06.628305 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:06.628273    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df446b3_d506_40c6_a7a3_78ebc9f20dbb.slice/crio-3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc WatchSource:0}: Error finding container 3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc: Status 404 returned error can't find the container with id 3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc
Nov 02 20:09:06.629146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.629039    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lcjhp-t56zv"]
Nov 02 20:09:06.989826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.989724    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"
Nov 02 20:09:06.989826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.989765    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv" event={"ID":"f6f00c3c-682c-406d-a845-44dc4baaa920","Type":"ContainerDied","Data":"6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b"}
Nov 02 20:09:06.990022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.989853    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e7fa013f5689ac7906c7140e370d40ed85f965cf95aa9aa19aa638a5cd8509b"
Nov 02 20:09:06.991475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.991448    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcjhp-t56zv" event={"ID":"4df446b3-d506-40c6-a7a3-78ebc9f20dbb","Type":"ContainerStarted","Data":"c589e10f974ecd496086def1776886a18449000999571c387c275b529075241c"}
Nov 02 20:09:06.991608 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:06.991491    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcjhp-t56zv" event={"ID":"4df446b3-d506-40c6-a7a3-78ebc9f20dbb","Type":"ContainerStarted","Data":"3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc"}
Nov 02 20:09:07.103552 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.103492    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lcjhp-t56zv" podStartSLOduration=2.103472825 podStartE2EDuration="2.103472825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:07.097913377 +0000 UTC m=+1035.972184290" watchObservedRunningTime="2026-11-02 20:09:07.103472825 +0000 UTC m=+1035.977743715"
Nov 02 20:09:07.151278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.151248    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:07.240139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.240056    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\" (UniqueName: \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\") pod \"95d3e622-6474-46a8-a673-5e0225c9a7d4\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") "
Nov 02 20:09:07.240590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.240148    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\") pod \"95d3e622-6474-46a8-a673-5e0225c9a7d4\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") "
Nov 02 20:09:07.240590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.240186    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\") pod \"95d3e622-6474-46a8-a673-5e0225c9a7d4\" (UID: \"95d3e622-6474-46a8-a673-5e0225c9a7d4\") "
Nov 02 20:09:07.240696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.240604    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca" pod "95d3e622-6474-46a8-a673-5e0225c9a7d4" (UID: "95d3e622-6474-46a8-a673-5e0225c9a7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:07.242680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.242658    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca" pod "95d3e622-6474-46a8-a673-5e0225c9a7d4" (UID: "95d3e622-6474-46a8-a673-5e0225c9a7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:07.242852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.242747    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf" pod "95d3e622-6474-46a8-a673-5e0225c9a7d4" (UID: "95d3e622-6474-46a8-a673-5e0225c9a7d4"). InnerVolumeSpecName "kube-api-access-gx4sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:07.341535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.341503    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:07.341535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.341531    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d3e622-6474-46a8-a673-5e0225c9a7d4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:07.341535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.341543    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4sf\" (UniqueName: \"kubernetes.io/projected/95d3e622-6474-46a8-a673-5e0225c9a7d4-kube-api-access-gx4sf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:07.999055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.999026    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"
Nov 02 20:09:07.999249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.999061    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv" event={"ID":"95d3e622-6474-46a8-a673-5e0225c9a7d4","Type":"ContainerDied","Data":"ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80"}
Nov 02 20:09:07.999249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:07.999119    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec97542f1d007b3bd838dc90e147e0a228b6e7ba2628dd7847af2188c921af80"
Nov 02 20:09:12.763081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:12.763048    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:12.775599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:12.775573    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:13.043183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:13.041379    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:09:15.176574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.176540    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:09:15.177092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177078    5099 cpu_manager.go:414] "containerMap: removing container" podUID="95d3e622-6474-46a8-a673-5e0225c9a7d4" containerName="revoke-keys"
Nov 02 20:09:15.177137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177092    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d3e622-6474-46a8-a673-5e0225c9a7d4" containerName="revoke-keys"
Nov 02 20:09:15.177137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177106    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f6f00c3c-682c-406d-a845-44dc4baaa920" containerName="revoke-keys"
Nov 02 20:09:15.177137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177111    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f00c3c-682c-406d-a845-44dc4baaa920" containerName="revoke-keys"
Nov 02 20:09:15.177678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177262    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d3e622-6474-46a8-a673-5e0225c9a7d4" containerName="revoke-keys"
Nov 02 20:09:15.177678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.177276    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f00c3c-682c-406d-a845-44dc4baaa920" containerName="revoke-keys"
Nov 02 20:09:15.181044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.181016    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.230629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.230596    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cc64d3-openshift-default-dockercfg-dq8fc\""
Nov 02 20:09:15.231080 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231060    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231107    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231131    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231169    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231290    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231314    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbrz\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231334    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231447    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.231528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.231508    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.281379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.281345    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:09:15.333108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333018    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333152    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbrz\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333187    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333247    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333323    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333349    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333372    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333446    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333493    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333531    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333624    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333675    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.333855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.333740    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.334035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.334010    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.335818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.335775    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.336036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.336013    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.386859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.386820    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.387012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.386948    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbrz\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.494750 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.494667    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:15.910562 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.910480    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:09:15.944254 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:15.944226    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89644dc_1644_4bec_89c1_131e2b4b988c.slice/crio-8d631665c5af623acce4011019becf2565e32f5f6bc6f2bdfd65e52454e0d0c6 WatchSource:0}: Error finding container 8d631665c5af623acce4011019becf2565e32f5f6bc6f2bdfd65e52454e0d0c6: Status 404 returned error can't find the container with id 8d631665c5af623acce4011019becf2565e32f5f6bc6f2bdfd65e52454e0d0c6
Nov 02 20:09:15.946418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.946386    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:15.946515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.946455    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:15.946515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:15.946482    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:09:16.055318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:16.055085    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" event={"ID":"a89644dc-1644-4bec-89c1-131e2b4b988c","Type":"ContainerStarted","Data":"8eaef030c6a79ee5c4e05ad2a22830b6659a3013a76ab504cfbd4d8f4a54fb8e"}
Nov 02 20:09:16.055318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:16.055146    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" event={"ID":"a89644dc-1644-4bec-89c1-131e2b4b988c","Type":"ContainerStarted","Data":"8d631665c5af623acce4011019becf2565e32f5f6bc6f2bdfd65e52454e0d0c6"}
Nov 02 20:09:16.093595 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:09:16.093559    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:09:16.166849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:16.166727    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podStartSLOduration=1.166707453 podStartE2EDuration="1.166707453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:16.164830796 +0000 UTC m=+1045.039101684" watchObservedRunningTime="2026-11-02 20:09:16.166707453 +0000 UTC m=+1045.040978344"
Nov 02 20:09:16.495221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:16.495144    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:17.497443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:17.497402    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 20:09:17.497856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:17.497457    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 20:09:18.495837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.495768    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 20:09:18.496036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.495860    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 20:09:18.554313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.554271    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"]
Nov 02 20:09:18.564616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.564576    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.594015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.593984    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"]
Nov 02 20:09:18.662114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.662073    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:18.662114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.662106    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:09:18.689685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.689652    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.689866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.689696    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.689866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.689818    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.790909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.790827    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.790909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.790873    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.791094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.790913    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.791602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.791569    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.793320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.793292    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.866410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.866380    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\") pod \"maas-api-revoke-sub-6680050c-7wl6w\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:18.878328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:18.878302    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:19.272897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:19.272863    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"]
Nov 02 20:09:19.274662 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:19.274636    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68793c29_8023_467c_9541_4a7eb73792df.slice/crio-9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a WatchSource:0}: Error finding container 9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a: Status 404 returned error can't find the container with id 9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a
Nov 02 20:09:19.495523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:19.495481    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 20:09:19.495671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:19.495552    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 20:09:20.089943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.089860    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w" event={"ID":"68793c29-8023-467c-9541-4a7eb73792df","Type":"ContainerStarted","Data":"8d5394459a4397687d28d77cbc70ea2358f5b5c021265fcb90a91b76fe4145e4"}
Nov 02 20:09:20.089943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.089909    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w" event={"ID":"68793c29-8023-467c-9541-4a7eb73792df","Type":"ContainerStarted","Data":"9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a"}
Nov 02 20:09:20.199582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.199520    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w" podStartSLOduration=2.199505967 podStartE2EDuration="2.199505967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:20.198459031 +0000 UTC m=+1049.072729934" watchObservedRunningTime="2026-11-02 20:09:20.199505967 +0000 UTC m=+1049.073776854"
Nov 02 20:09:20.496006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.495964    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 20:09:20.496189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.496026    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 20:09:20.726739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.726662    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"]
Nov 02 20:09:20.729717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.729697    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.783765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.783735    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"]
Nov 02 20:09:20.816701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.816651    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.816912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.816889    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.817041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.817023    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbj6\" (UniqueName: \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.917651 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.917613    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.917851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.917705    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.917851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.917761    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbj6\" (UniqueName: \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.918452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.918423    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.920124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.920100    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:20.972148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:20.972116    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbj6\" (UniqueName: \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\") pod \"maas-api-revoke-sub-c67b6247-r7vz5\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:21.063565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.063454    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:21.098569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.098532    5099 generic.go:356] "Generic (PLEG): container finished" podID="68793c29-8023-467c-9541-4a7eb73792df" containerID="8d5394459a4397687d28d77cbc70ea2358f5b5c021265fcb90a91b76fe4145e4" exitCode=0
Nov 02 20:09:21.099055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.098621    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w" event={"ID":"68793c29-8023-467c-9541-4a7eb73792df","Type":"ContainerDied","Data":"8d5394459a4397687d28d77cbc70ea2358f5b5c021265fcb90a91b76fe4145e4"}
Nov 02 20:09:21.432376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.432349    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"]
Nov 02 20:09:21.433352 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:21.433323    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b171bf4_2035_43d6_833e_1efab58caf0d.slice/crio-a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df WatchSource:0}: Error finding container a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df: Status 404 returned error can't find the container with id a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df
Nov 02 20:09:21.495957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.495929    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 20:09:21.496087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:21.495999    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 20:09:22.107957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.107853    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5" event={"ID":"7b171bf4-2035-43d6-833e-1efab58caf0d","Type":"ContainerStarted","Data":"9ff820f2482b2be2e2c79cfb466479b8bccac916cde7ad811205da5c53cf34a8"}
Nov 02 20:09:22.107957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.107932    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5" event={"ID":"7b171bf4-2035-43d6-833e-1efab58caf0d","Type":"ContainerStarted","Data":"a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df"}
Nov 02 20:09:22.303168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.303134    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:22.413635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.413572    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5" podStartSLOduration=2.413554183 podStartE2EDuration="2.413554183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:22.220553283 +0000 UTC m=+1051.094824169" watchObservedRunningTime="2026-11-02 20:09:22.413554183 +0000 UTC m=+1051.287825073"
Nov 02 20:09:22.436709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.436678    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\") pod \"68793c29-8023-467c-9541-4a7eb73792df\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") "
Nov 02 20:09:22.436918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.436844    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\") pod \"68793c29-8023-467c-9541-4a7eb73792df\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") "
Nov 02 20:09:22.436986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.436942    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\") pod \"68793c29-8023-467c-9541-4a7eb73792df\" (UID: \"68793c29-8023-467c-9541-4a7eb73792df\") "
Nov 02 20:09:22.437193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.437160    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca" pod "68793c29-8023-467c-9541-4a7eb73792df" (UID: "68793c29-8023-467c-9541-4a7eb73792df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:22.438244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.438222    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68793c29-8023-467c-9541-4a7eb73792df-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:22.439880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.439837    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca" pod "68793c29-8023-467c-9541-4a7eb73792df" (UID: "68793c29-8023-467c-9541-4a7eb73792df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:22.465984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.465938    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc" pod "68793c29-8023-467c-9541-4a7eb73792df" (UID: "68793c29-8023-467c-9541-4a7eb73792df"). InnerVolumeSpecName "kube-api-access-hhzmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:22.540082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.540039    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/68793c29-8023-467c-9541-4a7eb73792df-kube-api-access-hhzmc\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:22.540082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.540085    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68793c29-8023-467c-9541-4a7eb73792df-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:22.635690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.635642    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:22.636061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.636038    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:22.637376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:22.637327    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:09:23.115214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.115186    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"
Nov 02 20:09:23.115650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.115225    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w" event={"ID":"68793c29-8023-467c-9541-4a7eb73792df","Type":"ContainerDied","Data":"9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a"}
Nov 02 20:09:23.115650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.115274    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bea5431c0f68a43b02cac764726975267aaa74d25bbc524a3663d83a7e3f47a"
Nov 02 20:09:23.116918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.116895    5099 generic.go:356] "Generic (PLEG): container finished" podID="7b171bf4-2035-43d6-833e-1efab58caf0d" containerID="9ff820f2482b2be2e2c79cfb466479b8bccac916cde7ad811205da5c53cf34a8" exitCode=0
Nov 02 20:09:23.117043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.116978    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5" event={"ID":"7b171bf4-2035-43d6-833e-1efab58caf0d","Type":"ContainerDied","Data":"9ff820f2482b2be2e2c79cfb466479b8bccac916cde7ad811205da5c53cf34a8"}
Nov 02 20:09:23.212316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.212239    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"]
Nov 02 20:09:23.212846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.212831    5099 cpu_manager.go:414] "containerMap: removing container" podUID="68793c29-8023-467c-9541-4a7eb73792df" containerName="revoke-keys"
Nov 02 20:09:23.212898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.212846    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68793c29-8023-467c-9541-4a7eb73792df" containerName="revoke-keys"
Nov 02 20:09:23.212944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.212934    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="68793c29-8023-467c-9541-4a7eb73792df" containerName="revoke-keys"
Nov 02 20:09:23.215139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.215117    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.268309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.268273    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"]
Nov 02 20:09:23.348672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.348634    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.348856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.348707    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44q89\" (UniqueName: \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.348961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.348946    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.450740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.450700    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.450925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.450838    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.450925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.450909    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44q89\" (UniqueName: \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.451590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.451563    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.453271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.453243    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.504525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.504453    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44q89\" (UniqueName: \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\") pod \"maas-api-revoke-sub-05b97716-7f4tj\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.527828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.527768    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:23.902842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:23.902810    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"]
Nov 02 20:09:23.907342 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:23.907313    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ca2e4c_d9a4_4c10_b64f_8eea013762b6.slice/crio-9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3 WatchSource:0}: Error finding container 9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3: Status 404 returned error can't find the container with id 9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3
Nov 02 20:09:24.123996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.123960    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj" event={"ID":"32ca2e4c-d9a4-4c10-b64f-8eea013762b6","Type":"ContainerStarted","Data":"9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3"}
Nov 02 20:09:24.294812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.294772    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:24.360401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.360357    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\") pod \"7b171bf4-2035-43d6-833e-1efab58caf0d\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") "
Nov 02 20:09:24.360573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.360557    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\" (UniqueName: \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\") pod \"7b171bf4-2035-43d6-833e-1efab58caf0d\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") "
Nov 02 20:09:24.360630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.360604    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\") pod \"7b171bf4-2035-43d6-833e-1efab58caf0d\" (UID: \"7b171bf4-2035-43d6-833e-1efab58caf0d\") "
Nov 02 20:09:24.361018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.360995    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca" pod "7b171bf4-2035-43d6-833e-1efab58caf0d" (UID: "7b171bf4-2035-43d6-833e-1efab58caf0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:24.361150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.361127    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:24.362401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.362380    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca" pod "7b171bf4-2035-43d6-833e-1efab58caf0d" (UID: "7b171bf4-2035-43d6-833e-1efab58caf0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:24.363024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.363005    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6" pod "7b171bf4-2035-43d6-833e-1efab58caf0d" (UID: "7b171bf4-2035-43d6-833e-1efab58caf0d"). InnerVolumeSpecName "kube-api-access-xpbj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:24.462484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.462400    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b171bf4-2035-43d6-833e-1efab58caf0d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:24.462484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:24.462430    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbj6\" (UniqueName: \"kubernetes.io/projected/7b171bf4-2035-43d6-833e-1efab58caf0d-kube-api-access-xpbj6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:25.132041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:25.132008    5099 generic.go:356] "Generic (PLEG): container finished" podID="32ca2e4c-d9a4-4c10-b64f-8eea013762b6" containerID="a867cf005e7118a0de6ba4386a888afe71bff4e71c6647f70dfe44b589f60e93" exitCode=0
Nov 02 20:09:25.132463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:25.132099    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj" event={"ID":"32ca2e4c-d9a4-4c10-b64f-8eea013762b6","Type":"ContainerDied","Data":"a867cf005e7118a0de6ba4386a888afe71bff4e71c6647f70dfe44b589f60e93"}
Nov 02 20:09:25.133623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:25.133604    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"
Nov 02 20:09:25.133623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:25.133610    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5" event={"ID":"7b171bf4-2035-43d6-833e-1efab58caf0d","Type":"ContainerDied","Data":"a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df"}
Nov 02 20:09:25.133806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:25.133640    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3948a55a24a0925ea76c0832ba765deec73d1b3c0c78b82da82f127d91b59df"
Nov 02 20:09:26.454337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.453997    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:26.600166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.600123    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\") pod \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") "
Nov 02 20:09:26.600510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.600489    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\") pod \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") "
Nov 02 20:09:26.600648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.600634    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\" (UniqueName: \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\") pod \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\" (UID: \"32ca2e4c-d9a4-4c10-b64f-8eea013762b6\") "
Nov 02 20:09:26.609330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.608978    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca" pod "32ca2e4c-d9a4-4c10-b64f-8eea013762b6" (UID: "32ca2e4c-d9a4-4c10-b64f-8eea013762b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:26.622022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.621984    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89" pod "32ca2e4c-d9a4-4c10-b64f-8eea013762b6" (UID: "32ca2e4c-d9a4-4c10-b64f-8eea013762b6"). InnerVolumeSpecName "kube-api-access-44q89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:26.651376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.651276    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca" pod "32ca2e4c-d9a4-4c10-b64f-8eea013762b6" (UID: "32ca2e4c-d9a4-4c10-b64f-8eea013762b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:26.712647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.712601    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:26.712647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.712647    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44q89\" (UniqueName: \"kubernetes.io/projected/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-kube-api-access-44q89\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:26.712933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:26.712661    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ca2e4c-d9a4-4c10-b64f-8eea013762b6-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:27.164021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:27.163985    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj" event={"ID":"32ca2e4c-d9a4-4c10-b64f-8eea013762b6","Type":"ContainerDied","Data":"9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3"}
Nov 02 20:09:27.164319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:27.164287    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c40ef2100f3c6b5423da99c2280b2f454028c2453c2fb5cf3fdc4392befabc3"
Nov 02 20:09:27.164557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:27.164539    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"
Nov 02 20:09:31.503951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.503915    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"]
Nov 02 20:09:31.504480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504466    5099 cpu_manager.go:414] "containerMap: removing container" podUID="32ca2e4c-d9a4-4c10-b64f-8eea013762b6" containerName="revoke-keys"
Nov 02 20:09:31.504520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504481    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ca2e4c-d9a4-4c10-b64f-8eea013762b6" containerName="revoke-keys"
Nov 02 20:09:31.504520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504499    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7b171bf4-2035-43d6-833e-1efab58caf0d" containerName="revoke-keys"
Nov 02 20:09:31.504520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504505    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b171bf4-2035-43d6-833e-1efab58caf0d" containerName="revoke-keys"
Nov 02 20:09:31.504625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504618    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ca2e4c-d9a4-4c10-b64f-8eea013762b6" containerName="revoke-keys"
Nov 02 20:09:31.504660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.504630    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b171bf4-2035-43d6-833e-1efab58caf0d" containerName="revoke-keys"
Nov 02 20:09:31.507572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.507552    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.560502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.560468    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"]
Nov 02 20:09:31.581690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.581650    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.581901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.581883    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.582039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.582011    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72wn\" (UniqueName: \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.603058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.603033    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:09:31.603058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.603032    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:31.682771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.682736    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.683002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.682883    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.683002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.682956    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72wn\" (UniqueName: \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.683617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.683582    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.685751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.685729    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.737123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.737090    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72wn\" (UniqueName: \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\") pod \"maas-api-revoke-sub-6665dcd4-vthj2\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:31.821120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:31.821038    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:32.188522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.188492    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"]
Nov 02 20:09:32.190145 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:32.190104    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563f5405_288d_49e2_a436_d000fddd2e81.slice/crio-d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9 WatchSource:0}: Error finding container d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9: Status 404 returned error can't find the container with id d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9
Nov 02 20:09:32.201036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.201006    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2" event={"ID":"563f5405-288d-49e2-a436-d000fddd2e81","Type":"ContainerStarted","Data":"d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9"}
Nov 02 20:09:32.240435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.240409    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"]
Nov 02 20:09:32.244277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.244258    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.297335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.297296    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"]
Nov 02 20:09:32.391284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.391250    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.391456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.391309    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9zr\" (UniqueName: \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.391456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.391434    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.492383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.492348    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.492568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.492400    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9zr\" (UniqueName: \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.492568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.492458    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.493205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.493186    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.494671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.494649    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.546041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.546018    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9zr\" (UniqueName: \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\") pod \"maas-api-revoke-sub-ec65fcaa-xpp2g\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.557175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.557156    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:32.922645 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:32.922613    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02406e10_023c_4d34_aaec_ec3fd2ddf34c.slice/crio-1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1 WatchSource:0}: Error finding container 1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1: Status 404 returned error can't find the container with id 1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1
Nov 02 20:09:32.923048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:32.923026    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"]
Nov 02 20:09:33.210510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:33.210426    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g" event={"ID":"02406e10-023c-4d34-aaec-ec3fd2ddf34c","Type":"ContainerStarted","Data":"1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1"}
Nov 02 20:09:33.216636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:33.216599    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2" event={"ID":"563f5405-288d-49e2-a436-d000fddd2e81","Type":"ContainerStarted","Data":"deb6a0a2158d29d02f31cc65eb6a04898c9f69762a4d35c99d348cc5a22a8335"}
Nov 02 20:09:33.324095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:33.324044    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2" podStartSLOduration=2.324030265 podStartE2EDuration="2.324030265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:33.323674923 +0000 UTC m=+1062.197945813" watchObservedRunningTime="2026-11-02 20:09:33.324030265 +0000 UTC m=+1062.198301153"
Nov 02 20:09:34.226665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:34.226633    5099 generic.go:356] "Generic (PLEG): container finished" podID="02406e10-023c-4d34-aaec-ec3fd2ddf34c" containerID="01af96e1a291b1561b1924bfdc54960d998f5f5bd5e63e19381e362b32a799ee" exitCode=0
Nov 02 20:09:34.227167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:34.226719    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g" event={"ID":"02406e10-023c-4d34-aaec-ec3fd2ddf34c","Type":"ContainerDied","Data":"01af96e1a291b1561b1924bfdc54960d998f5f5bd5e63e19381e362b32a799ee"}
Nov 02 20:09:34.228579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:34.228552    5099 generic.go:356] "Generic (PLEG): container finished" podID="563f5405-288d-49e2-a436-d000fddd2e81" containerID="deb6a0a2158d29d02f31cc65eb6a04898c9f69762a4d35c99d348cc5a22a8335" exitCode=0
Nov 02 20:09:34.228765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:34.228604    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2" event={"ID":"563f5405-288d-49e2-a436-d000fddd2e81","Type":"ContainerDied","Data":"deb6a0a2158d29d02f31cc65eb6a04898c9f69762a4d35c99d348cc5a22a8335"}
Nov 02 20:09:35.490857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.490822    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:35.531018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.530989    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:35.647222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.647161    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\") pod \"563f5405-288d-49e2-a436-d000fddd2e81\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") "
Nov 02 20:09:35.656264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.656231    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\") pod \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") "
Nov 02 20:09:35.656511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.656492    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\" (UniqueName: \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\") pod \"563f5405-288d-49e2-a436-d000fddd2e81\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") "
Nov 02 20:09:35.656624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.656607    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\") pod \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") "
Nov 02 20:09:35.656703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.656661    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\") pod \"563f5405-288d-49e2-a436-d000fddd2e81\" (UID: \"563f5405-288d-49e2-a436-d000fddd2e81\") "
Nov 02 20:09:35.656764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.656714    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\" (UniqueName: \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\") pod \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\" (UID: \"02406e10-023c-4d34-aaec-ec3fd2ddf34c\") "
Nov 02 20:09:35.658167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.658141    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca" pod "563f5405-288d-49e2-a436-d000fddd2e81" (UID: "563f5405-288d-49e2-a436-d000fddd2e81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:35.658950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.658920    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca" pod "02406e10-023c-4d34-aaec-ec3fd2ddf34c" (UID: "02406e10-023c-4d34-aaec-ec3fd2ddf34c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:35.660095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.660069    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn" pod "563f5405-288d-49e2-a436-d000fddd2e81" (UID: "563f5405-288d-49e2-a436-d000fddd2e81"). InnerVolumeSpecName "kube-api-access-x72wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:35.661510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.661484    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca" pod "563f5405-288d-49e2-a436-d000fddd2e81" (UID: "563f5405-288d-49e2-a436-d000fddd2e81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:35.661621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.661554    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr" pod "02406e10-023c-4d34-aaec-ec3fd2ddf34c" (UID: "02406e10-023c-4d34-aaec-ec3fd2ddf34c"). InnerVolumeSpecName "kube-api-access-jt9zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:35.661621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.661558    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca" pod "02406e10-023c-4d34-aaec-ec3fd2ddf34c" (UID: "02406e10-023c-4d34-aaec-ec3fd2ddf34c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:35.758523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758432    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:35.758523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758477    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72wn\" (UniqueName: \"kubernetes.io/projected/563f5405-288d-49e2-a436-d000fddd2e81-kube-api-access-x72wn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:35.758523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758494    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02406e10-023c-4d34-aaec-ec3fd2ddf34c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:35.758523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758508    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:35.758523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758522    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt9zr\" (UniqueName: \"kubernetes.io/projected/02406e10-023c-4d34-aaec-ec3fd2ddf34c-kube-api-access-jt9zr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:35.758774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:35.758537    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563f5405-288d-49e2-a436-d000fddd2e81-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:36.265949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.265922    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"
Nov 02 20:09:36.266154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.265960    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2" event={"ID":"563f5405-288d-49e2-a436-d000fddd2e81","Type":"ContainerDied","Data":"d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9"}
Nov 02 20:09:36.266154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.266019    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b480a6094d2337dfbe4c3d38b5637a05e279b420e344fdfc240958b1083ca9"
Nov 02 20:09:36.267678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.267658    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"
Nov 02 20:09:36.267828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.267679    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g" event={"ID":"02406e10-023c-4d34-aaec-ec3fd2ddf34c","Type":"ContainerDied","Data":"1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1"}
Nov 02 20:09:36.267828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:36.267721    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1371e0def909f7b717148e67bcff8ea9d36f74d0fcfab50e38456a1684acd2a1"
Nov 02 20:09:46.095165 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:09:46.095121    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:09:47.084857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.084825    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"]
Nov 02 20:09:47.085469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085450    5099 cpu_manager.go:414] "containerMap: removing container" podUID="563f5405-288d-49e2-a436-d000fddd2e81" containerName="revoke-keys"
Nov 02 20:09:47.085469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085468    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563f5405-288d-49e2-a436-d000fddd2e81" containerName="revoke-keys"
Nov 02 20:09:47.085592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085484    5099 cpu_manager.go:414] "containerMap: removing container" podUID="02406e10-023c-4d34-aaec-ec3fd2ddf34c" containerName="revoke-keys"
Nov 02 20:09:47.085592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085490    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02406e10-023c-4d34-aaec-ec3fd2ddf34c" containerName="revoke-keys"
Nov 02 20:09:47.085657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085604    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="563f5405-288d-49e2-a436-d000fddd2e81" containerName="revoke-keys"
Nov 02 20:09:47.085657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.085629    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="02406e10-023c-4d34-aaec-ec3fd2ddf34c" containerName="revoke-keys"
Nov 02 20:09:47.089267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.089242    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.140183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.140147    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"]
Nov 02 20:09:47.183307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.183281    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:09:47.183443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.183401    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:09:47.213599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.213568    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.213768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.213752    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.213850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.213796    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6d5\" (UniqueName: \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.315209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.315173    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.315403 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.315214    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6d5\" (UniqueName: \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.315403 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.315282    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.315954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.315930    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.317616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.317596    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.369067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.369035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6d5\" (UniqueName: \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\") pod \"maas-api-revoke-sub-b328f656-8qfp7\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.401734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.401707    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:47.767509 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:47.767474    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb9950_109f_406f_990f_5ea2387c5e43.slice/crio-f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff WatchSource:0}: Error finding container f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff: Status 404 returned error can't find the container with id f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff
Nov 02 20:09:47.767663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.767579    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"]
Nov 02 20:09:47.940885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.940853    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"]
Nov 02 20:09:47.949126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.949100    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:47.997408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:47.997380    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"]
Nov 02 20:09:48.022104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.022000    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.022255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.022152    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wdw\" (UniqueName: \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.022255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.022235    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.123240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.123203    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.123387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.123258    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wdw\" (UniqueName: \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.123387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.123328    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.123864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.123844    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.125739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.125715    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.177078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.177047    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wdw\" (UniqueName: \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\") pod \"maas-api-revoke-sub-a68d95df-dfj2w\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.262027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.261990    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:48.353815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.353754    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7" event={"ID":"dbeb9950-109f-406f-990f-5ea2387c5e43","Type":"ContainerStarted","Data":"9ad4e8492ee1f9f75bebd5998798572116d5e32ce71fd4422a8da86d54eaefec"}
Nov 02 20:09:48.353982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.353825    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7" event={"ID":"dbeb9950-109f-406f-990f-5ea2387c5e43","Type":"ContainerStarted","Data":"f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff"}
Nov 02 20:09:48.462738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.462684    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7" podStartSLOduration=1.462671166 podStartE2EDuration="1.462671166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:48.461934632 +0000 UTC m=+1077.336205521" watchObservedRunningTime="2026-11-02 20:09:48.462671166 +0000 UTC m=+1077.336942054"
Nov 02 20:09:48.632677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:48.632647    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"]
Nov 02 20:09:48.633314 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:09:48.633278    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7b7068_d11b_4670_be25_96a1d1916267.slice/crio-91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad WatchSource:0}: Error finding container 91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad: Status 404 returned error can't find the container with id 91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad
Nov 02 20:09:49.361274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:49.361239    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w" event={"ID":"0b7b7068-d11b-4670-be25-96a1d1916267","Type":"ContainerStarted","Data":"e06d4df4dfa14cef9f72b0797ec2243a7f36b1cd7df76d6e97959e7185a57910"}
Nov 02 20:09:49.361776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:49.361293    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w" event={"ID":"0b7b7068-d11b-4670-be25-96a1d1916267","Type":"ContainerStarted","Data":"91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad"}
Nov 02 20:09:49.362758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:49.362733    5099 generic.go:356] "Generic (PLEG): container finished" podID="dbeb9950-109f-406f-990f-5ea2387c5e43" containerID="9ad4e8492ee1f9f75bebd5998798572116d5e32ce71fd4422a8da86d54eaefec" exitCode=0
Nov 02 20:09:49.362895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:49.362811    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7" event={"ID":"dbeb9950-109f-406f-990f-5ea2387c5e43","Type":"ContainerDied","Data":"9ad4e8492ee1f9f75bebd5998798572116d5e32ce71fd4422a8da86d54eaefec"}
Nov 02 20:09:49.469907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:49.469854    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w" podStartSLOduration=2.469838771 podStartE2EDuration="2.469838771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:49.467762821 +0000 UTC m=+1078.342033709" watchObservedRunningTime="2026-11-02 20:09:49.469838771 +0000 UTC m=+1078.344109659"
Nov 02 20:09:50.374025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.373991    5099 generic.go:356] "Generic (PLEG): container finished" podID="0b7b7068-d11b-4670-be25-96a1d1916267" containerID="e06d4df4dfa14cef9f72b0797ec2243a7f36b1cd7df76d6e97959e7185a57910" exitCode=0
Nov 02 20:09:50.374496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.374067    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w" event={"ID":"0b7b7068-d11b-4670-be25-96a1d1916267","Type":"ContainerDied","Data":"e06d4df4dfa14cef9f72b0797ec2243a7f36b1cd7df76d6e97959e7185a57910"}
Nov 02 20:09:50.538072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.538047    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:50.658074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.657989    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\") pod \"dbeb9950-109f-406f-990f-5ea2387c5e43\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") "
Nov 02 20:09:50.658074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.658037    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\" (UniqueName: \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\") pod \"dbeb9950-109f-406f-990f-5ea2387c5e43\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") "
Nov 02 20:09:50.658316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.658172    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\") pod \"dbeb9950-109f-406f-990f-5ea2387c5e43\" (UID: \"dbeb9950-109f-406f-990f-5ea2387c5e43\") "
Nov 02 20:09:50.658411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.658381    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca" pod "dbeb9950-109f-406f-990f-5ea2387c5e43" (UID: "dbeb9950-109f-406f-990f-5ea2387c5e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:50.658694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.658677    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:50.660203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.660176    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5" pod "dbeb9950-109f-406f-990f-5ea2387c5e43" (UID: "dbeb9950-109f-406f-990f-5ea2387c5e43"). InnerVolumeSpecName "kube-api-access-vm6d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:50.660555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.660535    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca" pod "dbeb9950-109f-406f-990f-5ea2387c5e43" (UID: "dbeb9950-109f-406f-990f-5ea2387c5e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:50.760198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.760151    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6d5\" (UniqueName: \"kubernetes.io/projected/dbeb9950-109f-406f-990f-5ea2387c5e43-kube-api-access-vm6d5\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:50.760198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:50.760190    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbeb9950-109f-406f-990f-5ea2387c5e43-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:51.382891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.382860    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"
Nov 02 20:09:51.383296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.382893    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7" event={"ID":"dbeb9950-109f-406f-990f-5ea2387c5e43","Type":"ContainerDied","Data":"f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff"}
Nov 02 20:09:51.383296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.382939    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f267ba77614959d9aed318df3353061114ece2a2278c3aeba7afe95038b490ff"
Nov 02 20:09:51.542122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.542096    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:09:51.671137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.671102    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\") pod \"0b7b7068-d11b-4670-be25-96a1d1916267\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") "
Nov 02 20:09:51.671335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.671200    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\") pod \"0b7b7068-d11b-4670-be25-96a1d1916267\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") "
Nov 02 20:09:51.671335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.671255    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\" (UniqueName: \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\") pod \"0b7b7068-d11b-4670-be25-96a1d1916267\" (UID: \"0b7b7068-d11b-4670-be25-96a1d1916267\") "
Nov 02 20:09:51.672385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.672350    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca" pod "0b7b7068-d11b-4670-be25-96a1d1916267" (UID: "0b7b7068-d11b-4670-be25-96a1d1916267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:09:51.674244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.674217    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca" pod "0b7b7068-d11b-4670-be25-96a1d1916267" (UID: "0b7b7068-d11b-4670-be25-96a1d1916267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:09:51.677222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.677196    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw" pod "0b7b7068-d11b-4670-be25-96a1d1916267" (UID: "0b7b7068-d11b-4670-be25-96a1d1916267"). InnerVolumeSpecName "kube-api-access-b8wdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:09:51.772605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.772571    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:51.772605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.772601    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7b7068-d11b-4670-be25-96a1d1916267-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:51.772605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:51.772610    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wdw\" (UniqueName: \"kubernetes.io/projected/0b7b7068-d11b-4670-be25-96a1d1916267-kube-api-access-b8wdw\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:09:52.392268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:52.392229    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w" event={"ID":"0b7b7068-d11b-4670-be25-96a1d1916267","Type":"ContainerDied","Data":"91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad"}
Nov 02 20:09:52.392656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:52.392292    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91f6bcbd28358358a1cfc8637550ca45ea1388916ed660a392be1b6b1ff6a7ad"
Nov 02 20:09:52.392656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:09:52.392248    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"
Nov 02 20:10:01.917056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:01.917023    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:10:01.917563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:01.917224    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy" containerID="cri-o://58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d" gracePeriod=30
Nov 02 20:10:02.209222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.209140    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.209753    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0b7b7068-d11b-4670-be25-96a1d1916267" containerName="revoke-keys"
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.209769    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7b7068-d11b-4670-be25-96a1d1916267" containerName="revoke-keys"
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.209863    5099 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb9950-109f-406f-990f-5ea2387c5e43" containerName="revoke-keys"
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.209872    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb9950-109f-406f-990f-5ea2387c5e43" containerName="revoke-keys"
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.210003    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb9950-109f-406f-990f-5ea2387c5e43" containerName="revoke-keys"
Nov 02 20:10:02.210251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.210023    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7b7068-d11b-4670-be25-96a1d1916267" containerName="revoke-keys"
Nov 02 20:10:02.212122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.212105    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.262968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.262935    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"openshift-service-ca.crt\""
Nov 02 20:10:02.263196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.263022    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"kube-root-ca.crt\""
Nov 02 20:10:02.263196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.263093    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"default-dockercfg-gpd7p\""
Nov 02 20:10:02.263343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.263242    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"distinct-w2-9fcedd-kserve-self-signed-certs\""
Nov 02 20:10:02.266802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.266757    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:10:02.307887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.307851    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.308058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.308016    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.308118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.308101    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.308181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.308165    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8cl6\" (UniqueName: \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.308243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.308227    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.308291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.308280    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.409737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.409698    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.409993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.409766    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.409993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.409843    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8cl6\" (UniqueName: \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.409993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.409876    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.409993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.409961    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.410240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.410022    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.410240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.410202    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.410530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.410508    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.410683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.410665    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.412484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.412457    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.412632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.412615    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.469837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.469746    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8cl6\" (UniqueName: \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\") pod \"distinct-w2-9fcedd-kserve-5886f586fb-szd9r\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.525199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.525166    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:02.935096 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:10:02.935065    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0cffea3_f787_48a0_ab5e_d02f85984911.slice/crio-9cd6a90e2449ae317b1ead008684ab60045d037e16a69b0561b79bb4db02556a WatchSource:0}: Error finding container 9cd6a90e2449ae317b1ead008684ab60045d037e16a69b0561b79bb4db02556a: Status 404 returned error can't find the container with id 9cd6a90e2449ae317b1ead008684ab60045d037e16a69b0561b79bb4db02556a
Nov 02 20:10:02.935437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:02.935156    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:10:03.478235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:03.478196    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" event={"ID":"c0cffea3-f787-48a0-ab5e-d02f85984911","Type":"ContainerStarted","Data":"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"}
Nov 02 20:10:03.478235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:03.478246    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" event={"ID":"c0cffea3-f787-48a0-ab5e-d02f85984911","Type":"ContainerStarted","Data":"9cd6a90e2449ae317b1ead008684ab60045d037e16a69b0561b79bb4db02556a"}
Nov 02 20:10:03.525289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:03.525259    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:03.585478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:03.585426    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podStartSLOduration=1.5854110700000001 podStartE2EDuration="1.58541107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:03.584597358 +0000 UTC m=+1092.458868247" watchObservedRunningTime="2026-11-02 20:10:03.58541107 +0000 UTC m=+1092.459681958"
Nov 02 20:10:05.475817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.475767    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:10:05.480377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.480355    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.530553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.530526    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"distinct-w3-2052e3-kserve-self-signed-certs\""
Nov 02 20:10:05.534451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.534418    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:10:05.650373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650321    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.650373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650386    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7ld\" (UniqueName: \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.650662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650506    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.650662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650574    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.650662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650624    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.650879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.650666    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752039    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752113    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752147    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752177    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752244    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752267    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7ld\" (UniqueName: \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752602    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752613    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.752838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.752658    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.754250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.754230    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.754603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.754582    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:05.805583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:05.805553    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7ld\" (UniqueName: \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\") pod \"distinct-w3-2052e3-kserve-6866dff59-6p5vp\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:06.094480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:06.094403    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:06.467975 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:10:06.467931    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15125abf_f6ee_4a64_b3c8_305fe354d020.slice/crio-7f2bac11ede1ff65541f4b831eb245e7530170671b7fe4619649f2de86a37608 WatchSource:0}: Error finding container 7f2bac11ede1ff65541f4b831eb245e7530170671b7fe4619649f2de86a37608: Status 404 returned error can't find the container with id 7f2bac11ede1ff65541f4b831eb245e7530170671b7fe4619649f2de86a37608
Nov 02 20:10:06.469232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:06.469198    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:10:06.502648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:06.502608    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" event={"ID":"15125abf-f6ee-4a64-b3c8-305fe354d020","Type":"ContainerStarted","Data":"7f2bac11ede1ff65541f4b831eb245e7530170671b7fe4619649f2de86a37608"}
Nov 02 20:10:07.227952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.227928    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:10:07.289049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289013    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289058    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289089    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289127    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289183    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289283    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289318    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289376    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289459    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:07.289627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289474    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\") pod \"86df5e11-0409-4087-b289-899149ffab93\" (UID: \"86df5e11-0409-4087-b289-899149ffab93\") "
Nov 02 20:10:07.289627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289586    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:10:07.289818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289695    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:07.289959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289763    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:07.289959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:07.289959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289956    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86df5e11-0409-4087-b289-899149ffab93-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.290087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289981    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.290087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.289995    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.291532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.291512    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:07.291620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.291601    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:10:07.291824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.291803    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "kube-api-access-s2lm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:10:07.291915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.291898    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token" pod "86df5e11-0409-4087-b289-899149ffab93" (UID: "86df5e11-0409-4087-b289-899149ffab93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:10:07.390599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390557    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.390599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390588    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2lm6\" (UniqueName: \"kubernetes.io/projected/86df5e11-0409-4087-b289-899149ffab93-kube-api-access-s2lm6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.390599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390599    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.390599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390607    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.390988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390615    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86df5e11-0409-4087-b289-899149ffab93-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.390988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.390623    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86df5e11-0409-4087-b289-899149ffab93-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:07.511111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.511003    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" event={"ID":"15125abf-f6ee-4a64-b3c8-305fe354d020","Type":"ContainerStarted","Data":"4786637c6673084008e09c223673b7423a9270bf54b163b49ba8fdc45e46545e"}
Nov 02 20:10:07.512644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.512537    5099 generic.go:356] "Generic (PLEG): container finished" podID="86df5e11-0409-4087-b289-899149ffab93" containerID="58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d" exitCode=0
Nov 02 20:10:07.512644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.512600    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"
Nov 02 20:10:07.512644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.512637    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" event={"ID":"86df5e11-0409-4087-b289-899149ffab93","Type":"ContainerDied","Data":"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"}
Nov 02 20:10:07.512958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.512693    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd" event={"ID":"86df5e11-0409-4087-b289-899149ffab93","Type":"ContainerDied","Data":"9a2ddda2acf2f17dc8b8a1aebb4e60020c61fd6ce81221376c4253f86466625e"}
Nov 02 20:10:07.512958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.512725    5099 scope.go:122] "RemoveContainer" containerID="58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"
Nov 02 20:10:07.523661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.523634    5099 scope.go:122] "RemoveContainer" containerID="58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"
Nov 02 20:10:07.523943 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:10:07.523925    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d\": container with ID starting with 58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d not found: ID does not exist" containerID="58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"
Nov 02 20:10:07.523996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.523951    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d"} err="failed to get container status \"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d\": rpc error: code = NotFound desc = could not find container \"58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d\": container with ID starting with 58ef2a1f779f0f97a0ac79708270583adafdf7d68b5d998c9f20e5c78c6fd01d not found: ID does not exist"
Nov 02 20:10:07.619755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.619705    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podStartSLOduration=2.619690526 podStartE2EDuration="2.619690526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:07.618447058 +0000 UTC m=+1096.492717946" watchObservedRunningTime="2026-11-02 20:10:07.619690526 +0000 UTC m=+1096.493961414"
Nov 02 20:10:07.725940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.725897    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:10:07.756540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:07.756495    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-wb4xd"]
Nov 02 20:10:08.095450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:08.095414    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:09.263091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:09.263052    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86df5e11-0409-4087-b289-899149ffab93" path="/var/lib/kubelet/pods/86df5e11-0409-4087-b289-899149ffab93/volumes"
Nov 02 20:10:12.525558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:12.525517    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:12.541265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:12.541231    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:12.558834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:12.558801    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:10:16.094595 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:10:16.094557    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:10:16.095053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:16.094616    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:16.107685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:16.107658    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:16.587689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:16.587655    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:10:31.078274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.078201    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:10:31.078804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.078773    5099 cpu_manager.go:414] "containerMap: removing container" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy"
Nov 02 20:10:31.078851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.078806    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy"
Nov 02 20:10:31.078927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.078917    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="86df5e11-0409-4087-b289-899149ffab93" containerName="istio-proxy"
Nov 02 20:10:31.084149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.084124    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.133560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.133528    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-982b8a-openshift-default-dockercfg-lsv7m\""
Nov 02 20:10:31.184686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.184653    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:10:31.203828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.203797    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.203861    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.203943    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.203989    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.204088    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.204126    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.204158    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92vd\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.204215    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.204382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.204357    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305334    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305429    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305495    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305519    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305550    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305606    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305849    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.305959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305913    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92vd\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305961    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.305989    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.306055    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.306264    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.306469    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.306622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.306583    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.308430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.308408    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.308522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.308493    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.360940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.360915    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92vd\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.361096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.360914    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.400974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.400939    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:31.847386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.847360    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:10:31.847712 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:10:31.847693    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700f978f_63c3_4947_b7b6_c925cea7d1fb.slice/crio-7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48 WatchSource:0}: Error finding container 7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48: Status 404 returned error can't find the container with id 7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48
Nov 02 20:10:31.850085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.850048    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:10:31.850199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.850125    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:10:31.850260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:31.850196    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:10:32.719644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:32.719577    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" event={"ID":"700f978f-63c3-4947-b7b6-c925cea7d1fb","Type":"ContainerStarted","Data":"657550c75d17009eb5308a9d50686b2cc1dc771205a46b726ac80061a89f297b"}
Nov 02 20:10:32.720182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:32.719688    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" event={"ID":"700f978f-63c3-4947-b7b6-c925cea7d1fb","Type":"ContainerStarted","Data":"7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48"}
Nov 02 20:10:32.932902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:32.932833    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podStartSLOduration=1.932813197 podStartE2EDuration="1.932813197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:32.882342096 +0000 UTC m=+1121.756612997" watchObservedRunningTime="2026-11-02 20:10:32.932813197 +0000 UTC m=+1121.807084086"
Nov 02 20:10:33.401313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:33.401277    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:33.402702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:33.402677    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:33.402831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:33.402730    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:34.402071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:34.402037    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:34.402470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:34.402087    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:35.401703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:35.401674    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:35.402014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:35.401877    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:36.404157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:36.404112    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:36.404752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:36.404717    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:37.402081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:37.402041    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:37.402427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:37.402102    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:38.401926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:38.401887    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 20:10:38.402305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:38.401955    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 20:10:39.406913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:39.406882    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:39.407409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:39.407205    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:39.408008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:39.407990    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:10:43.854037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:43.853998    5099 generic.go:356] "Generic (PLEG): container finished" podID="4df446b3-d506-40c6-a7a3-78ebc9f20dbb" containerID="c589e10f974ecd496086def1776886a18449000999571c387c275b529075241c" exitCode=0
Nov 02 20:10:43.854437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:43.854072    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcjhp-t56zv" event={"ID":"4df446b3-d506-40c6-a7a3-78ebc9f20dbb","Type":"ContainerDied","Data":"c589e10f974ecd496086def1776886a18449000999571c387c275b529075241c"}
Nov 02 20:10:45.031692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.031665    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:10:45.078424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078386    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\" (UniqueName: \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\") pod \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") "
Nov 02 20:10:45.078610 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078435    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\") pod \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") "
Nov 02 20:10:45.078610 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078480    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\") pod \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") "
Nov 02 20:10:45.078610 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078531    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\") pod \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\" (UID: \"4df446b3-d506-40c6-a7a3-78ebc9f20dbb\") "
Nov 02 20:10:45.078910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078886    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path" pod "4df446b3-d506-40c6-a7a3-78ebc9f20dbb" (UID: "4df446b3-d506-40c6-a7a3-78ebc9f20dbb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:10:45.079011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078943    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle" pod "4df446b3-d506-40c6-a7a3-78ebc9f20dbb" (UID: "4df446b3-d506-40c6-a7a3-78ebc9f20dbb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:10:45.079011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.078958    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle" pod "4df446b3-d506-40c6-a7a3-78ebc9f20dbb" (UID: "4df446b3-d506-40c6-a7a3-78ebc9f20dbb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:10:45.079208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.079175    5099 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-service-ca-bundle\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:45.079208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.079206    5099 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-archives-path\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:45.079395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.079220    5099 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-trusted-ca-bundle\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:45.080846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.080821    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht" pod "4df446b3-d506-40c6-a7a3-78ebc9f20dbb" (UID: "4df446b3-d506-40c6-a7a3-78ebc9f20dbb"). InnerVolumeSpecName "kube-api-access-qkdht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:10:45.180168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.180079    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdht\" (UniqueName: \"kubernetes.io/projected/4df446b3-d506-40c6-a7a3-78ebc9f20dbb-kube-api-access-qkdht\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:10:45.868925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.868894    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lcjhp-t56zv"
Nov 02 20:10:45.868925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.868909    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lcjhp-t56zv" event={"ID":"4df446b3-d506-40c6-a7a3-78ebc9f20dbb","Type":"ContainerDied","Data":"3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc"}
Nov 02 20:10:45.869131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:45.868951    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a28cfe9600b2dd9b88710d03fa1430a416a3ee3aad486aeaabc482560cf22dc"
Nov 02 20:10:46.097211 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:10:46.097169    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:10:57.892119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.892082    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:10:57.892620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.892607    5099 cpu_manager.go:414] "containerMap: removing container" podUID="4df446b3-d506-40c6-a7a3-78ebc9f20dbb" containerName="insights-gathering"
Nov 02 20:10:57.892664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.892621    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df446b3-d506-40c6-a7a3-78ebc9f20dbb" containerName="insights-gathering"
Nov 02 20:10:57.892768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.892754    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df446b3-d506-40c6-a7a3-78ebc9f20dbb" containerName="insights-gathering"
Nov 02 20:10:57.895715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.895696    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:57.951727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.951689    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:10:57.991933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.991904    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"openshift-service-ca.crt\""
Nov 02 20:10:57.991933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.991913    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"kube-root-ca.crt\""
Nov 02 20:10:57.999223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.999202    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"default-dockercfg-sxn9n\""
Nov 02 20:10:57.999340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:57.999223    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"distinct-w0-035e47-kserve-self-signed-certs\""
Nov 02 20:10:58.030495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030456    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxfm\" (UniqueName: \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.030653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030568    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.030653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030617    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.030809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030655    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.030809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030708    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.030925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.030904    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131599    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131643    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxfm\" (UniqueName: \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131677    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131695    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131730    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.131750    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.132146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.132075    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.132146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.132093    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.132249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.132187    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.133965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.133941    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.134250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.134230    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.186205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.186140    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxfm\" (UniqueName: \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\") pod \"distinct-w0-035e47-kserve-765c7dbfff-xdlgb\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.207051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.207028    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:10:58.582198 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:10:58.582165    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881614d6_c2ab_4039_b7b5_7230392dfda8.slice/crio-1e993b6baa839d45858a0cacbdac3f03da33bfc5bf997418f068e0538267b540 WatchSource:0}: Error finding container 1e993b6baa839d45858a0cacbdac3f03da33bfc5bf997418f068e0538267b540: Status 404 returned error can't find the container with id 1e993b6baa839d45858a0cacbdac3f03da33bfc5bf997418f068e0538267b540
Nov 02 20:10:58.582373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.582346    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:10:58.979621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.979587    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" event={"ID":"881614d6-c2ab-4039-b7b5-7230392dfda8","Type":"ContainerStarted","Data":"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"}
Nov 02 20:10:58.980119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:58.979637    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" event={"ID":"881614d6-c2ab-4039-b7b5-7230392dfda8","Type":"ContainerStarted","Data":"1e993b6baa839d45858a0cacbdac3f03da33bfc5bf997418f068e0538267b540"}
Nov 02 20:10:59.087134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:59.087085    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podStartSLOduration=2.087071072 podStartE2EDuration="2.087071072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:59.086913181 +0000 UTC m=+1147.961184070" watchObservedRunningTime="2026-11-02 20:10:59.087071072 +0000 UTC m=+1147.961341959"
Nov 02 20:10:59.218275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:10:59.217619    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:11:08.207294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:08.207261    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:11:08.220248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:08.220216    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:11:09.062950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:09.062921    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:11:16.098596 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:11:16.098561    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:11:16.127995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.127966    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:11:16.132630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.132612    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176389    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176474    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176532    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176554    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176689    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176741    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176814    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176892    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.176976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.176954    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz6c\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.234565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.234533    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:11:16.277545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277504    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277597    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz6c\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277692    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277722    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277777    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277829    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.277897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277874    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277918    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.277946    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.278186    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.278220    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.278349    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.278466    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.278991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.278966    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.280069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.280050    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.280183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.280165    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.331600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.331565    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz6c\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.331914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.331895    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.447668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.447626    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:16.869487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.869452    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:11:16.884194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.884152    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:16.884301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.884247    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:16.884301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:16.884287    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:17.111385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:17.111346    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" event={"ID":"ac482a80-0e41-4747-b414-dc0f1506789f","Type":"ContainerStarted","Data":"0cc5070f4285bb40706817fed7c953b9940dc811e9a7e90c28cf1dcebe7384e7"}
Nov 02 20:11:17.111959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:17.111409    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" event={"ID":"ac482a80-0e41-4747-b414-dc0f1506789f","Type":"ContainerStarted","Data":"ab9d5e207c73799f268da026a91e5113f90d96e2c0d6ef5d9eed3e131e6c5fad"}
Nov 02 20:11:17.452897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:17.452853    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:17.456962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:17.456916    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:17.457161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:17.456985    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:18.459806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:18.459214    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:18.459806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:18.459276    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:19.451265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:19.451224    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:19.451560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:19.451530    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:20.452806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:20.452159    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:20.452806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:20.452223    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:21.452806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:21.449147    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:21.452806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:21.449209    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:22.458249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:22.458207    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:22.458860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:22.458825    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:23.448498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:23.448457    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 20:11:23.448676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:23.448524    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 20:11:24.452748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:24.452718    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:24.453195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:24.453178    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:24.453934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:24.453915    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:11:24.610544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:24.610494    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podStartSLOduration=8.610482603 podStartE2EDuration="8.610482603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:17.224070354 +0000 UTC m=+1166.098341246" watchObservedRunningTime="2026-11-02 20:11:24.610482603 +0000 UTC m=+1173.484753490"
Nov 02 20:11:31.024312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.024278    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"]
Nov 02 20:11:31.031135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.031111    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.081151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.081122    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"]
Nov 02 20:11:31.123202 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.123170    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:11:31.123380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.123250    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:11:31.179856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.179825    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.180037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.179976    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.180102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.180085    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl785\" (UniqueName: \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.281871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.281758    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl785\" (UniqueName: \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.282022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.281921    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.282066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.282050    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.282588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.282558    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.284413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.284385    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.335362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.335334    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl785\" (UniqueName: \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\") pod \"maas-api-revoke-sub-0ca7551b-hc78l\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.346216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.346190    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:31.714378 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:11:31.714345    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2debddb_fc95_4f28_b340_22c2aba1de2c.slice/crio-bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047 WatchSource:0}: Error finding container bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047: Status 404 returned error can't find the container with id bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047
Nov 02 20:11:31.716197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:31.716170    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"]
Nov 02 20:11:32.257034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:32.257002    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l" event={"ID":"d2debddb-fc95-4f28-b340-22c2aba1de2c","Type":"ContainerStarted","Data":"d5a4166b5829df45ef91bced412f6b44ed77568741eeec9009ce8b2af7eb22d2"}
Nov 02 20:11:32.257458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:32.257050    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l" event={"ID":"d2debddb-fc95-4f28-b340-22c2aba1de2c","Type":"ContainerStarted","Data":"bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047"}
Nov 02 20:11:32.372481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:32.372422    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l" podStartSLOduration=2.372384908 podStartE2EDuration="2.372384908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:32.369879596 +0000 UTC m=+1181.244150484" watchObservedRunningTime="2026-11-02 20:11:32.372384908 +0000 UTC m=+1181.246655794"
Nov 02 20:11:33.292432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:33.292393    5099 generic.go:356] "Generic (PLEG): container finished" podID="d2debddb-fc95-4f28-b340-22c2aba1de2c" containerID="d5a4166b5829df45ef91bced412f6b44ed77568741eeec9009ce8b2af7eb22d2" exitCode=0
Nov 02 20:11:33.294376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:33.294347    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l" event={"ID":"d2debddb-fc95-4f28-b340-22c2aba1de2c","Type":"ContainerDied","Data":"d5a4166b5829df45ef91bced412f6b44ed77568741eeec9009ce8b2af7eb22d2"}
Nov 02 20:11:34.454401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.454376    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:34.629498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.629465    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\") pod \"d2debddb-fc95-4f28-b340-22c2aba1de2c\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") "
Nov 02 20:11:34.629653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.629517    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\") pod \"d2debddb-fc95-4f28-b340-22c2aba1de2c\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") "
Nov 02 20:11:34.629731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.629714    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\" (UniqueName: \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\") pod \"d2debddb-fc95-4f28-b340-22c2aba1de2c\" (UID: \"d2debddb-fc95-4f28-b340-22c2aba1de2c\") "
Nov 02 20:11:34.629912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.629889    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca" pod "d2debddb-fc95-4f28-b340-22c2aba1de2c" (UID: "d2debddb-fc95-4f28-b340-22c2aba1de2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:11:34.630224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.630202    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:34.631760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.631730    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785" pod "d2debddb-fc95-4f28-b340-22c2aba1de2c" (UID: "d2debddb-fc95-4f28-b340-22c2aba1de2c"). InnerVolumeSpecName "kube-api-access-hl785". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:11:34.631760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.631731    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca" pod "d2debddb-fc95-4f28-b340-22c2aba1de2c" (UID: "d2debddb-fc95-4f28-b340-22c2aba1de2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:11:34.731339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.731298    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl785\" (UniqueName: \"kubernetes.io/projected/d2debddb-fc95-4f28-b340-22c2aba1de2c-kube-api-access-hl785\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:34.731513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:34.731345    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2debddb-fc95-4f28-b340-22c2aba1de2c-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:35.310047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:35.310020    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"
Nov 02 20:11:35.310216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:35.310020    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l" event={"ID":"d2debddb-fc95-4f28-b340-22c2aba1de2c","Type":"ContainerDied","Data":"bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047"}
Nov 02 20:11:35.310216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:35.310142    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf187661cc5f3ddb8ad491cdff78f2bdea253c269297325b73149c4c389c6047"
Nov 02 20:11:45.475118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.475081    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"]
Nov 02 20:11:45.475777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.475657    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d2debddb-fc95-4f28-b340-22c2aba1de2c" containerName="revoke-keys"
Nov 02 20:11:45.475777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.475671    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2debddb-fc95-4f28-b340-22c2aba1de2c" containerName="revoke-keys"
Nov 02 20:11:45.475925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.475820    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2debddb-fc95-4f28-b340-22c2aba1de2c" containerName="revoke-keys"
Nov 02 20:11:45.479642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.479624    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.534670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.534634    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"]
Nov 02 20:11:45.575943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.575831    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:11:45.577011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.576857    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:11:45.589865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.589834    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxjs\" (UniqueName: \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.590117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.590092    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.590241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.590224    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.691755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.691725    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxjs\" (UniqueName: \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.691942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.691774    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.691942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.691817    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.692497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.692474    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.694473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.694443    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.746183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.746111    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxjs\" (UniqueName: \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\") pod \"maas-api-revoke-sub-a7fe5680-jm2hx\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:45.795056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:45.795022    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:46.095516 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:11:46.095406    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:11:46.168458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:46.168423    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"]
Nov 02 20:11:46.168628 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:11:46.168456    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf766afad_1b95_4cee_937b_e2352f7e930b.slice/crio-480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736 WatchSource:0}: Error finding container 480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736: Status 404 returned error can't find the container with id 480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736
Nov 02 20:11:46.399632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:46.399595    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx" event={"ID":"f766afad-1b95-4cee-937b-e2352f7e930b","Type":"ContainerStarted","Data":"480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736"}
Nov 02 20:11:46.955761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:46.955728    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:11:46.959912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:46.959891    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007475    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6c7k\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007624    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007698    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007735    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007757    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.007912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007894    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.008003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.007991    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.008037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.008025    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.008095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.008081    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.054560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.054525    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:11:47.109353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109319    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109434    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109492    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109519    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109567    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6c7k\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109638    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109695    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109738    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.109805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109766    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.110077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109841    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.110077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.109930    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.110183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.110157    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.110255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.110232    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.110390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.110374    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.112004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.111979    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.112349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.112330    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.163493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.163463    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.163712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.163688    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6c7k\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.241803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.241692    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:11:47.256537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.256499    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.277624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.277594    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:47.292877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.292847    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:11:47.311643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311615    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.311825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311675    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.311885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311836    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.311924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311885    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.311964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311942    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.311997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.311966    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdz2\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.312039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.312004    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.312095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.312049    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.312095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.312088    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.409527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.409489    5099 generic.go:356] "Generic (PLEG): container finished" podID="f766afad-1b95-4cee-937b-e2352f7e930b" containerID="1288411b0f52896b2188380c2bb0370e33d643bde0a20dfd36cb88108d91f345" exitCode=0
Nov 02 20:11:47.409716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.409632    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx" event={"ID":"f766afad-1b95-4cee-937b-e2352f7e930b","Type":"ContainerDied","Data":"1288411b0f52896b2188380c2bb0370e33d643bde0a20dfd36cb88108d91f345"}
Nov 02 20:11:47.413035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413010    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413055    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413127    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413173    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413244    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413269    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdz2\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413313    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413361    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413401    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.413625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413491    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.414093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413811    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.414093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.413832    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.414093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.414049    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.414342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.414308    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.415746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.415725    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.416560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.416540    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.467217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.467165    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.467439 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.467416    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdz2\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.574153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.574068    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:47.704921 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:11:47.704884    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2095e6d0_8120_4baa_82d0_79685a165272.slice/crio-88ad431950ef0c6e19304e0226206276a06e6b17b721c418b47d14ca7ff7f94a WatchSource:0}: Error finding container 88ad431950ef0c6e19304e0226206276a06e6b17b721c418b47d14ca7ff7f94a: Status 404 returned error can't find the container with id 88ad431950ef0c6e19304e0226206276a06e6b17b721c418b47d14ca7ff7f94a
Nov 02 20:11:47.706583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.706554    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:11:47.708682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.708638    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:47.708792 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.708733    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:47.708846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.708795    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:47.964840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.964762    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:11:47.965271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.965003    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:47.965271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.965091    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:47.965271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:47.965133    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:11:48.420276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.420238    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" event={"ID":"53d9649a-9e29-4297-9f23-e6c800efba87","Type":"ContainerStarted","Data":"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"}
Nov 02 20:11:48.420467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.420306    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" event={"ID":"53d9649a-9e29-4297-9f23-e6c800efba87","Type":"ContainerStarted","Data":"6d25fcdb49ba59cc8711d839b8a76deeaad9299dfc8f10d21df91dd944de0289"}
Nov 02 20:11:48.422572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.422543    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" event={"ID":"2095e6d0-8120-4baa-82d0-79685a165272","Type":"ContainerStarted","Data":"aca6070f3e74b6179847a0d886af75cf6b83ba4c4e080f31ad407e7a42946b61"}
Nov 02 20:11:48.422711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.422590    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" event={"ID":"2095e6d0-8120-4baa-82d0-79685a165272","Type":"ContainerStarted","Data":"88ad431950ef0c6e19304e0226206276a06e6b17b721c418b47d14ca7ff7f94a"}
Nov 02 20:11:48.581808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.578174    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:48.622926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.622884    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:48.659807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.659045    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podStartSLOduration=1.6590217790000001 podStartE2EDuration="1.659021779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:48.541261264 +0000 UTC m=+1197.415532153" watchObservedRunningTime="2026-11-02 20:11:48.659021779 +0000 UTC m=+1197.533292667"
Nov 02 20:11:48.735353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.735108    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\") pod \"f766afad-1b95-4cee-937b-e2352f7e930b\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") "
Nov 02 20:11:48.735594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.735570    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca" pod "f766afad-1b95-4cee-937b-e2352f7e930b" (UID: "f766afad-1b95-4cee-937b-e2352f7e930b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:11:48.735674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.735660    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\" (UniqueName: \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\") pod \"f766afad-1b95-4cee-937b-e2352f7e930b\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") "
Nov 02 20:11:48.735739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.735726    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\") pod \"f766afad-1b95-4cee-937b-e2352f7e930b\" (UID: \"f766afad-1b95-4cee-937b-e2352f7e930b\") "
Nov 02 20:11:48.741806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.740957    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:48.784050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.783992    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca" pod "f766afad-1b95-4cee-937b-e2352f7e930b" (UID: "f766afad-1b95-4cee-937b-e2352f7e930b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:11:48.786836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.785995    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs" pod "f766afad-1b95-4cee-937b-e2352f7e930b" (UID: "f766afad-1b95-4cee-937b-e2352f7e930b"). InnerVolumeSpecName "kube-api-access-ggxjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:11:48.806984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.797216    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podStartSLOduration=2.797190887 podStartE2EDuration="2.797190887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:48.68718584 +0000 UTC m=+1197.561456729" watchObservedRunningTime="2026-11-02 20:11:48.797190887 +0000 UTC m=+1197.671461776"
Nov 02 20:11:48.862817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.862529    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f766afad-1b95-4cee-937b-e2352f7e930b-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:48.862817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:48.862568    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxjs\" (UniqueName: \"kubernetes.io/projected/f766afad-1b95-4cee-937b-e2352f7e930b-kube-api-access-ggxjs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:11:49.299941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.299852    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:49.302874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.302829    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:49.303028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.302955    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:49.430757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.430723    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"
Nov 02 20:11:49.430965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.430759    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx" event={"ID":"f766afad-1b95-4cee-937b-e2352f7e930b","Type":"ContainerDied","Data":"480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736"}
Nov 02 20:11:49.430965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.430866    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="480b9fd270be22444de8c8caff29d5ee66e568285e2ecf8e47f72b3ecc87c736"
Nov 02 20:11:49.581863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.581436    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:49.581863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:49.581497    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:50.278226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:50.278186    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:50.278547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:50.278517    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:50.575274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:50.575182    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:50.576957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:50.576915    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:51.301804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.299887    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:51.301804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.299944    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:51.591869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.590970    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:51.591869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.591030    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:51.621757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.621726    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:11:51.626690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:51.626665    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:11:52.281806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:52.281225    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:52.281806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:52.281297    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:52.576268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:52.576170    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:52.576569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:52.576525    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:53.278766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:53.278729    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:53.279384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:53.279353    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:53.595001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:53.594906    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:53.595243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:53.595213    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:54.278119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:54.278080    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 20:11:54.278300 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:54.278154    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 20:11:54.575294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:54.575205    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 20:11:54.575294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:54.575269    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 20:11:55.283425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.283396    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:55.283834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.283808    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:55.284504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.284485    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:11:55.598152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.598004    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:55.598845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.598824    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:11:55.600130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:11:55.600105    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:12:01.414358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.414277    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:12:01.415007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.414989    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f766afad-1b95-4cee-937b-e2352f7e930b" containerName="revoke-keys"
Nov 02 20:12:01.415083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.415008    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f766afad-1b95-4cee-937b-e2352f7e930b" containerName="revoke-keys"
Nov 02 20:12:01.415157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.415145    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f766afad-1b95-4cee-937b-e2352f7e930b" containerName="revoke-keys"
Nov 02 20:12:01.419975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.419957    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488022    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488073    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488103    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488216    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488256    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488343    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488374    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488392    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwrh\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.488497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.488475    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589377    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589414    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwrh\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589441    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589515    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589706    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589792    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.589906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589865    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589908    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589909    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589954    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.589983    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.590174    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.590481    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.590601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.590497    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.592305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.592278    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.592446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.592426    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.604207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.604179    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:12:01.644013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.643983    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwrh\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.644177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.644096    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.733995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.733904    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:01.743839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.743807    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:12:01.748723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.748696    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792428    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792503    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792532    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792651    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792689    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792718    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792760    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792842    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmkb\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.792903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.792881    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.807762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.806413    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:12:01.894546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894514    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.894731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894552    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.894731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894576    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.894731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894606    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.894731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894649    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmkb\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.894731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894677    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.894757    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895004    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895043    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895049    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895017    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895266    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895472    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.895713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.895692    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.897065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.897048    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.897130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.897107    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.948416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.948387    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:01.948628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:01.948611    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmkb\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:02.078366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.078277    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:02.170421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.169753    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:12:02.174929 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:02.174896    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f787339_a01a_4321_9acc_e4a485ff36fd.slice/crio-b5915b3cf2fda21ce588ef88e0ce4f8d7a234e0f4d8816782f48602efd9fa659 WatchSource:0}: Error finding container b5915b3cf2fda21ce588ef88e0ce4f8d7a234e0f4d8816782f48602efd9fa659: Status 404 returned error can't find the container with id b5915b3cf2fda21ce588ef88e0ce4f8d7a234e0f4d8816782f48602efd9fa659
Nov 02 20:12:02.177725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.177692    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.178022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.177758    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.178022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.177815    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.448035 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:02.447992    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2f4829_1312_4341_aed0_9bd94add9887.slice/crio-2ad9d52cbab1913238d17877c0f2e1aaed1d194c961d13db0eb94175e48ec92a WatchSource:0}: Error finding container 2ad9d52cbab1913238d17877c0f2e1aaed1d194c961d13db0eb94175e48ec92a: Status 404 returned error can't find the container with id 2ad9d52cbab1913238d17877c0f2e1aaed1d194c961d13db0eb94175e48ec92a
Nov 02 20:12:02.449047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.449020    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:12:02.451283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.451242    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.451361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.451330    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.451407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.451371    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:12:02.654284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.654244    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" event={"ID":"8f787339-a01a-4321-9acc-e4a485ff36fd","Type":"ContainerStarted","Data":"e82158a28cf0dbbffa37e3a8de208d6e416480769f7cffd3c6e24d7035d871d2"}
Nov 02 20:12:02.654557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.654528    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" event={"ID":"8f787339-a01a-4321-9acc-e4a485ff36fd","Type":"ContainerStarted","Data":"b5915b3cf2fda21ce588ef88e0ce4f8d7a234e0f4d8816782f48602efd9fa659"}
Nov 02 20:12:02.656205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.656179    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" event={"ID":"1c2f4829-1312-4341-aed0-9bd94add9887","Type":"ContainerStarted","Data":"3dd834a808740464c1cbaec0873a4f1911f4d4dbee4a88a0269c01521d9d7bf4"}
Nov 02 20:12:02.656298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.656234    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" event={"ID":"1c2f4829-1312-4341-aed0-9bd94add9887","Type":"ContainerStarted","Data":"2ad9d52cbab1913238d17877c0f2e1aaed1d194c961d13db0eb94175e48ec92a"}
Nov 02 20:12:02.741996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.741899    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:02.882238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.882187    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podStartSLOduration=1.882173154 podStartE2EDuration="1.882173154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:02.877856204 +0000 UTC m=+1211.752127093" watchObservedRunningTime="2026-11-02 20:12:02.882173154 +0000 UTC m=+1211.756444041"
Nov 02 20:12:02.883329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:02.883081    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podStartSLOduration=1.883071509 podStartE2EDuration="1.883071509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:02.768119192 +0000 UTC m=+1211.642390081" watchObservedRunningTime="2026-11-02 20:12:02.883071509 +0000 UTC m=+1211.757342399"
Nov 02 20:12:03.080962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:03.080864    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:03.082162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:03.082126    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:03.082291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:03.082182    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:03.736082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:03.736049    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:03.736469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:03.736102    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:04.080862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:04.080198    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:04.080862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:04.080261    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:04.735179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:04.735150    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:04.735350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:04.735196    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:05.079866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:05.079200    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:05.079866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:05.079262    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:05.734328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:05.734288    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:05.734555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:05.734355    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:06.079241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:06.079147    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:06.079512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:06.079477    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:06.734694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:06.734656    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:06.735120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:06.734710    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:07.088864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:07.084202    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:07.088864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:07.084266    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:07.735195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:07.735160    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:07.735601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:07.735213    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:08.088866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:08.085222    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:08.088866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:08.085303    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:08.735104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:08.735064    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 20:12:08.735352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:08.735134    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 20:12:09.079040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.078940    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 20:12:09.079040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.079015    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 20:12:09.611074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.611033    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"]
Nov 02 20:12:09.617072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.617044    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.631646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.631610    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.631860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.631819    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgljb\" (UniqueName: \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.632014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.631994    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.667722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.667680    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"]
Nov 02 20:12:09.707823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.707764    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:12:09.714824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.708850    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:12:09.737808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.734068    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.737808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.734246    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.737808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.734393    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgljb\" (UniqueName: \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.737808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.736102    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.759214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.759156    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.775624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.775580    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:09.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.776361    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:09.777397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.777374    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:12:09.789147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.789114    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgljb\" (UniqueName: \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\") pod \"maas-api-revoke-sub-bda036ed-ct6xx\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:09.942452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:09.942353    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:10.100283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.100255    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:10.102379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.102267    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:10.104543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.104493    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:12:10.422004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.421969    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"]
Nov 02 20:12:10.422425 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:10.422399    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4abe0f_01c2_45fd_934f_b4c46d60a0c6.slice/crio-6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57 WatchSource:0}: Error finding container 6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57: Status 404 returned error can't find the container with id 6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57
Nov 02 20:12:10.774366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.773143    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx" event={"ID":"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6","Type":"ContainerStarted","Data":"3cf3301ae0c045263865a8c6bba2763ff3a326ebc838c76049abc8093ae69806"}
Nov 02 20:12:10.774366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.773252    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx" event={"ID":"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6","Type":"ContainerStarted","Data":"6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57"}
Nov 02 20:12:10.882512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:10.882457    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx" podStartSLOduration=1.8824445189999999 podStartE2EDuration="1.882444519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:10.880212592 +0000 UTC m=+1219.754483595" watchObservedRunningTime="2026-11-02 20:12:10.882444519 +0000 UTC m=+1219.756715406"
Nov 02 20:12:11.780725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:11.780631    5099 generic.go:356] "Generic (PLEG): container finished" podID="8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" containerID="3cf3301ae0c045263865a8c6bba2763ff3a326ebc838c76049abc8093ae69806" exitCode=0
Nov 02 20:12:11.781135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:11.780713    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx" event={"ID":"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6","Type":"ContainerDied","Data":"3cf3301ae0c045263865a8c6bba2763ff3a326ebc838c76049abc8093ae69806"}
Nov 02 20:12:13.007936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.007904    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:13.105346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.105233    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\" (UniqueName: \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\") pod \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") "
Nov 02 20:12:13.105529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.105359    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\") pod \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") "
Nov 02 20:12:13.105605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.105553    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\") pod \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\" (UID: \"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6\") "
Nov 02 20:12:13.106126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.106098    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca" pod "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" (UID: "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:13.108809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.108764    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca" pod "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" (UID: "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:13.108946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.108882    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb" pod "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" (UID: "8f4abe0f-01c2-45fd-934f-b4c46d60a0c6"). InnerVolumeSpecName "kube-api-access-xgljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:13.208043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.208002    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:13.208228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.208048    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgljb\" (UniqueName: \"kubernetes.io/projected/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-kube-api-access-xgljb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:13.208228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.208067    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:13.810130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.810097    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx" event={"ID":"8f4abe0f-01c2-45fd-934f-b4c46d60a0c6","Type":"ContainerDied","Data":"6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57"}
Nov 02 20:12:13.810309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.810153    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db23f5979f12010c6e4bc89f783277c44e537f785af1cf8e19c93e88970fd57"
Nov 02 20:12:13.810309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:13.810217    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"
Nov 02 20:12:16.094490 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:12:16.094445    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:12:25.270747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.270721    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"]
Nov 02 20:12:25.271798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.271754    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" containerName="revoke-keys"
Nov 02 20:12:25.271798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.271798    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" containerName="revoke-keys"
Nov 02 20:12:25.272013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.271980    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" containerName="revoke-keys"
Nov 02 20:12:25.277103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.277079    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.300389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.300362    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"]
Nov 02 20:12:25.342196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.342166    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:12:25.342364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.342212    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:12:25.424317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.424283    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.424495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.424351    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.424495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.424478    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65zq\" (UniqueName: \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.526224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.526138    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65zq\" (UniqueName: \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.526377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.526246    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.526377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.526283    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.527047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.527023    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.528588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.528564    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.580181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.580142    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65zq\" (UniqueName: \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\") pod \"maas-api-revoke-sub-ddccc649-pkmpz\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.592795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.592759    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:25.966413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:25.966380    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"]
Nov 02 20:12:25.966886 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:25.966850    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27544f53_d2fa_4f15_94dc_2893028ae92a.slice/crio-2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac WatchSource:0}: Error finding container 2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac: Status 404 returned error can't find the container with id 2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac
Nov 02 20:12:26.906226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:26.906187    5099 generic.go:356] "Generic (PLEG): container finished" podID="27544f53-d2fa-4f15-94dc-2893028ae92a" containerID="95ed2762dcfc0bb46bd6ee4efce0adc0c9390c9411f01fbd95d0b9a8dc4e5b01" exitCode=0
Nov 02 20:12:26.906589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:26.906263    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz" event={"ID":"27544f53-d2fa-4f15-94dc-2893028ae92a","Type":"ContainerDied","Data":"95ed2762dcfc0bb46bd6ee4efce0adc0c9390c9411f01fbd95d0b9a8dc4e5b01"}
Nov 02 20:12:26.906589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:26.906318    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz" event={"ID":"27544f53-d2fa-4f15-94dc-2893028ae92a","Type":"ContainerStarted","Data":"2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac"}
Nov 02 20:12:28.083974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.083949    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:28.268662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.268565    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\" (UniqueName: \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\") pod \"27544f53-d2fa-4f15-94dc-2893028ae92a\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") "
Nov 02 20:12:28.268930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.268719    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\") pod \"27544f53-d2fa-4f15-94dc-2893028ae92a\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") "
Nov 02 20:12:28.268930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.268831    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\") pod \"27544f53-d2fa-4f15-94dc-2893028ae92a\" (UID: \"27544f53-d2fa-4f15-94dc-2893028ae92a\") "
Nov 02 20:12:28.269088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.269063    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca" pod "27544f53-d2fa-4f15-94dc-2893028ae92a" (UID: "27544f53-d2fa-4f15-94dc-2893028ae92a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:28.269373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.269346    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:28.271693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.271671    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq" pod "27544f53-d2fa-4f15-94dc-2893028ae92a" (UID: "27544f53-d2fa-4f15-94dc-2893028ae92a"). InnerVolumeSpecName "kube-api-access-f65zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:28.272702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.272678    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca" pod "27544f53-d2fa-4f15-94dc-2893028ae92a" (UID: "27544f53-d2fa-4f15-94dc-2893028ae92a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:28.371256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.371220    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27544f53-d2fa-4f15-94dc-2893028ae92a-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:28.371455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.371415    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65zq\" (UniqueName: \"kubernetes.io/projected/27544f53-d2fa-4f15-94dc-2893028ae92a-kube-api-access-f65zq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:28.921993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.921963    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"
Nov 02 20:12:28.921993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.921977    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz" event={"ID":"27544f53-d2fa-4f15-94dc-2893028ae92a","Type":"ContainerDied","Data":"2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac"}
Nov 02 20:12:28.922218 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:28.922058    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf61f10439e82b79a7ed5b6b45499a250db637753da38451b6389239de5efac"
Nov 02 20:12:46.095198 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:12:46.095158    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:12:49.447239 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.447204    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"]
Nov 02 20:12:49.448036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.448013    5099 cpu_manager.go:414] "containerMap: removing container" podUID="27544f53-d2fa-4f15-94dc-2893028ae92a" containerName="revoke-keys"
Nov 02 20:12:49.448036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.448032    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27544f53-d2fa-4f15-94dc-2893028ae92a" containerName="revoke-keys"
Nov 02 20:12:49.448212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.448140    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="27544f53-d2fa-4f15-94dc-2893028ae92a" containerName="revoke-keys"
Nov 02 20:12:49.451458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.451441    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.503176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.503142    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"]
Nov 02 20:12:49.545048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.545020    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:12:49.545232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.545098    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:12:49.577344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.577314    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.577542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.577493    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4nt\" (UniqueName: \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.577616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.577589    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.678498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.678436    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.678714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.678576    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.678714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.678680    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4nt\" (UniqueName: \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.679339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.679313    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.681064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.681038    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.732533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.732462    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4nt\" (UniqueName: \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\") pod \"maas-api-revoke-sub-1861e62b-m6cj2\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:49.765414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:49.765388    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:50.131933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.131904    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"]
Nov 02 20:12:50.137621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.137594    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.137893 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:50.137862    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44997a9_d55b_4f4d_b810_022c3aa74935.slice/crio-5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee WatchSource:0}: Error finding container 5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee: Status 404 returned error can't find the container with id 5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee
Nov 02 20:12:50.139698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.139669    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"]
Nov 02 20:12:50.214373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.214332    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"]
Nov 02 20:12:50.310872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.310830    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.311048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.310972    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.311048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.311013    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z55g\" (UniqueName: \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.412244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.412151    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.412423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.412392    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.412518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.412477    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z55g\" (UniqueName: \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.412925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.412904    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.414707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.414689    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.466194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.466168    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z55g\" (UniqueName: \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\") pod \"maas-api-revoke-sub-e0f594a5-jfzqn\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.469022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.468998    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:50.837349 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:12:50.837294    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d72231b_3111_4dbb_968c_608cac4c2b3b.slice/crio-9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3 WatchSource:0}: Error finding container 9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3: Status 404 returned error can't find the container with id 9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3
Nov 02 20:12:50.840890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:50.840382    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"]
Nov 02 20:12:51.108254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:51.108169    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn" event={"ID":"3d72231b-3111-4dbb-968c-608cac4c2b3b","Type":"ContainerStarted","Data":"9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3"}
Nov 02 20:12:51.109848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:51.109821    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2" event={"ID":"d44997a9-d55b-4f4d-b810-022c3aa74935","Type":"ContainerStarted","Data":"b9d79cae0ed4a27ea97064c32a8f51c2168a40db67f2548c0f9779f4a952725a"}
Nov 02 20:12:51.109998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:51.109860    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2" event={"ID":"d44997a9-d55b-4f4d-b810-022c3aa74935","Type":"ContainerStarted","Data":"5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee"}
Nov 02 20:12:51.221005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:51.220954    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2" podStartSLOduration=2.220937948 podStartE2EDuration="2.220937948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:51.217586182 +0000 UTC m=+1260.091857072" watchObservedRunningTime="2026-11-02 20:12:51.220937948 +0000 UTC m=+1260.095208836"
Nov 02 20:12:52.117710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:52.117675    5099 generic.go:356] "Generic (PLEG): container finished" podID="3d72231b-3111-4dbb-968c-608cac4c2b3b" containerID="f65d9d91de66cb73bd58c39c3d1846094aa376995f74b1183946f78c1f72d1f4" exitCode=0
Nov 02 20:12:52.118153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:52.117760    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn" event={"ID":"3d72231b-3111-4dbb-968c-608cac4c2b3b","Type":"ContainerDied","Data":"f65d9d91de66cb73bd58c39c3d1846094aa376995f74b1183946f78c1f72d1f4"}
Nov 02 20:12:52.119418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:52.119396    5099 generic.go:356] "Generic (PLEG): container finished" podID="d44997a9-d55b-4f4d-b810-022c3aa74935" containerID="b9d79cae0ed4a27ea97064c32a8f51c2168a40db67f2548c0f9779f4a952725a" exitCode=0
Nov 02 20:12:52.119537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:52.119477    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2" event={"ID":"d44997a9-d55b-4f4d-b810-022c3aa74935","Type":"ContainerDied","Data":"b9d79cae0ed4a27ea97064c32a8f51c2168a40db67f2548c0f9779f4a952725a"}
Nov 02 20:12:53.310439 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.310411    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:53.342822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.340256    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:53.362977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.362944    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\") pod \"3d72231b-3111-4dbb-968c-608cac4c2b3b\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") "
Nov 02 20:12:53.363117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363019    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\" (UniqueName: \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\") pod \"d44997a9-d55b-4f4d-b810-022c3aa74935\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") "
Nov 02 20:12:53.363117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363061    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\" (UniqueName: \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\") pod \"3d72231b-3111-4dbb-968c-608cac4c2b3b\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") "
Nov 02 20:12:53.363117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363093    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\") pod \"d44997a9-d55b-4f4d-b810-022c3aa74935\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") "
Nov 02 20:12:53.363301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363141    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\") pod \"3d72231b-3111-4dbb-968c-608cac4c2b3b\" (UID: \"3d72231b-3111-4dbb-968c-608cac4c2b3b\") "
Nov 02 20:12:53.363301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363229    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\") pod \"d44997a9-d55b-4f4d-b810-022c3aa74935\" (UID: \"d44997a9-d55b-4f4d-b810-022c3aa74935\") "
Nov 02 20:12:53.363414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363385    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca" pod "3d72231b-3111-4dbb-968c-608cac4c2b3b" (UID: "3d72231b-3111-4dbb-968c-608cac4c2b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:53.363513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363484    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca" pod "d44997a9-d55b-4f4d-b810-022c3aa74935" (UID: "d44997a9-d55b-4f4d-b810-022c3aa74935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:12:53.363836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363812    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:53.363933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.363840    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:53.365538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.365514    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g" pod "3d72231b-3111-4dbb-968c-608cac4c2b3b" (UID: "3d72231b-3111-4dbb-968c-608cac4c2b3b"). InnerVolumeSpecName "kube-api-access-4z55g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:53.366042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.366021    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt" pod "d44997a9-d55b-4f4d-b810-022c3aa74935" (UID: "d44997a9-d55b-4f4d-b810-022c3aa74935"). InnerVolumeSpecName "kube-api-access-5s4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:12:53.366117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.366022    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca" pod "d44997a9-d55b-4f4d-b810-022c3aa74935" (UID: "d44997a9-d55b-4f4d-b810-022c3aa74935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:53.366117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.366062    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca" pod "3d72231b-3111-4dbb-968c-608cac4c2b3b" (UID: "3d72231b-3111-4dbb-968c-608cac4c2b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:12:53.464836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.464771    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4nt\" (UniqueName: \"kubernetes.io/projected/d44997a9-d55b-4f4d-b810-022c3aa74935-kube-api-access-5s4nt\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:53.464836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.464839    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z55g\" (UniqueName: \"kubernetes.io/projected/3d72231b-3111-4dbb-968c-608cac4c2b3b-kube-api-access-4z55g\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:53.465072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.464855    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d72231b-3111-4dbb-968c-608cac4c2b3b-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:53.465072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:53.464869    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44997a9-d55b-4f4d-b810-022c3aa74935-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:12:54.135500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.135474    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"
Nov 02 20:12:54.135664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.135479    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn" event={"ID":"3d72231b-3111-4dbb-968c-608cac4c2b3b","Type":"ContainerDied","Data":"9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3"}
Nov 02 20:12:54.135664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.135602    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d675615d8722aa5f2f2d4f219dcb322be66f41431be2b5a42177033be5fffd3"
Nov 02 20:12:54.136903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.136876    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"
Nov 02 20:12:54.137005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.136918    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2" event={"ID":"d44997a9-d55b-4f4d-b810-022c3aa74935","Type":"ContainerDied","Data":"5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee"}
Nov 02 20:12:54.137005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:12:54.136969    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fef02f680a0b91315fa6c7c48e96c3b59b169f0417a38e534669cd33882b5ee"
Nov 02 20:13:01.919808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:01.918918    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:02.040814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.039913    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:02.072807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.071892    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:02.555808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.554930    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:13:02.587826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.586880    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:02.913808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.912948    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:13:02.913808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.913077    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:02.920933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.920859    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:13:02.921951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:02.921918    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:03.049981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.049923    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:03.050461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.050426    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:03.136837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.136798    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:03.137718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.137697    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:03.230160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.230029    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:03.521881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.519883    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:03.559831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.529426    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:03.559831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.530259    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:03.559831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.559056    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:03.559831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.559468    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:03.560287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.560264    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:03.694576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:03.694524    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:05.621682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.621641    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"]
Nov 02 20:13:05.622261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622240    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3d72231b-3111-4dbb-968c-608cac4c2b3b" containerName="revoke-keys"
Nov 02 20:13:05.622261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622256    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d72231b-3111-4dbb-968c-608cac4c2b3b" containerName="revoke-keys"
Nov 02 20:13:05.622426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622286    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d44997a9-d55b-4f4d-b810-022c3aa74935" containerName="revoke-keys"
Nov 02 20:13:05.622426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622291    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44997a9-d55b-4f4d-b810-022c3aa74935" containerName="revoke-keys"
Nov 02 20:13:05.622426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622405    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d72231b-3111-4dbb-968c-608cac4c2b3b" containerName="revoke-keys"
Nov 02 20:13:05.622426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.622417    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44997a9-d55b-4f4d-b810-022c3aa74935" containerName="revoke-keys"
Nov 02 20:13:05.627456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.627437    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.679561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.679532    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"]
Nov 02 20:13:05.715521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.715483    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.715711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.715620    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.715711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.715665    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkxz\" (UniqueName: \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.720230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.720205    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:05.720365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.720238    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:05.817383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.817351    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.817583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.817421    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.817583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.817458    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkxz\" (UniqueName: \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.818593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.818562    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.820053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.820035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.871959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.871872    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkxz\" (UniqueName: \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\") pod \"maas-api-revoke-sub-78dcdf6a-v29xr\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:05.941821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:05.941761    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:06.076335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:06.076301    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"]
Nov 02 20:13:06.082893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:06.082819    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7b8aaee-d8nfh"]
Nov 02 20:13:06.315541 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:06.315506    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1feb72d_ca9a_4491_920f_59062f272917.slice/crio-8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a WatchSource:0}: Error finding container 8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a: Status 404 returned error can't find the container with id 8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a
Nov 02 20:13:06.317190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:06.317168    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"]
Nov 02 20:13:06.548295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:06.548129    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr" event={"ID":"c1feb72d-ca9a-4491-920f-59062f272917","Type":"ContainerStarted","Data":"8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a"}
Nov 02 20:13:07.260052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:07.260013    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79caf93d-a4e3-4a4d-a733-b223d2f6ec4a" path="/var/lib/kubelet/pods/79caf93d-a4e3-4a4d-a733-b223d2f6ec4a/volumes"
Nov 02 20:13:07.559290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:07.559200    5099 generic.go:356] "Generic (PLEG): container finished" podID="c1feb72d-ca9a-4491-920f-59062f272917" containerID="9eec75596014a1b71fa98b0f02e4f7ee3bae6ed747c6e1051e475b1015868c46" exitCode=0
Nov 02 20:13:07.559468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:07.559287    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr" event={"ID":"c1feb72d-ca9a-4491-920f-59062f272917","Type":"ContainerDied","Data":"9eec75596014a1b71fa98b0f02e4f7ee3bae6ed747c6e1051e475b1015868c46"}
Nov 02 20:13:08.736751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.736728    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:08.853459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.853347    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\") pod \"c1feb72d-ca9a-4491-920f-59062f272917\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") "
Nov 02 20:13:08.854150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.854127    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\" (UniqueName: \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\") pod \"c1feb72d-ca9a-4491-920f-59062f272917\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") "
Nov 02 20:13:08.854762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.854744    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\") pod \"c1feb72d-ca9a-4491-920f-59062f272917\" (UID: \"c1feb72d-ca9a-4491-920f-59062f272917\") "
Nov 02 20:13:08.855434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.855307    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca" pod "c1feb72d-ca9a-4491-920f-59062f272917" (UID: "c1feb72d-ca9a-4491-920f-59062f272917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:08.856000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.855982    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:08.858345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.858316    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca" pod "c1feb72d-ca9a-4491-920f-59062f272917" (UID: "c1feb72d-ca9a-4491-920f-59062f272917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:08.859238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.859215    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz" pod "c1feb72d-ca9a-4491-920f-59062f272917" (UID: "c1feb72d-ca9a-4491-920f-59062f272917"). InnerVolumeSpecName "kube-api-access-5kkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:08.957967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.957935    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1feb72d-ca9a-4491-920f-59062f272917-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:08.957967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:08.957962    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkxz\" (UniqueName: \"kubernetes.io/projected/c1feb72d-ca9a-4491-920f-59062f272917-kube-api-access-5kkxz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:09.579147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:09.579070    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"
Nov 02 20:13:09.579147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:09.579084    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr" event={"ID":"c1feb72d-ca9a-4491-920f-59062f272917","Type":"ContainerDied","Data":"8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a"}
Nov 02 20:13:09.579147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:09.579138    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d3a76595b40ec7bf0cd9c6ecd8c466169855ad865c74201c573c4753948868a"
Nov 02 20:13:11.259020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.258991    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"]
Nov 02 20:13:11.259526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.259512    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c1feb72d-ca9a-4491-920f-59062f272917" containerName="revoke-keys"
Nov 02 20:13:11.259571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.259527    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1feb72d-ca9a-4491-920f-59062f272917" containerName="revoke-keys"
Nov 02 20:13:11.259663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.259654    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1feb72d-ca9a-4491-920f-59062f272917" containerName="revoke-keys"
Nov 02 20:13:11.262260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.262240    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.307504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.307471    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"]
Nov 02 20:13:11.315129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.311965    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:11.349036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.349002    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:11.406382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.406349    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.406558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.406409    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsmx\" (UniqueName: \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.406889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.406862    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.508545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.508495    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.508709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.508605    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.508709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.508657    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsmx\" (UniqueName: \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.509330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.509263    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.511018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.510995    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.562855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.562821    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsmx\" (UniqueName: \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\") pod \"maas-api-revoke-sub-f2ae6f89-2skkd\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.575140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.575111    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:11.945637 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:11.945602    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaec533d_5e2d_4c29_8d60_1ab86165e48d.slice/crio-8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779 WatchSource:0}: Error finding container 8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779: Status 404 returned error can't find the container with id 8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779
Nov 02 20:13:11.947923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:11.947894    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"]
Nov 02 20:13:12.606474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:12.606384    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd" event={"ID":"caec533d-5e2d-4c29-8d60-1ab86165e48d","Type":"ContainerStarted","Data":"775e4078b1b20cb3e9c11a220c2863b3feca8478f4eed8979e2f2913d0cbfa6b"}
Nov 02 20:13:12.606860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:12.606482    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd" event={"ID":"caec533d-5e2d-4c29-8d60-1ab86165e48d","Type":"ContainerStarted","Data":"8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779"}
Nov 02 20:13:12.719135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:12.719077    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd" podStartSLOduration=1.719064763 podStartE2EDuration="1.719064763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:12.715583788 +0000 UTC m=+1281.589854670" watchObservedRunningTime="2026-11-02 20:13:12.719064763 +0000 UTC m=+1281.593335668"
Nov 02 20:13:13.615091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:13.615054    5099 generic.go:356] "Generic (PLEG): container finished" podID="caec533d-5e2d-4c29-8d60-1ab86165e48d" containerID="775e4078b1b20cb3e9c11a220c2863b3feca8478f4eed8979e2f2913d0cbfa6b" exitCode=0
Nov 02 20:13:13.615518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:13.615139    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd" event={"ID":"caec533d-5e2d-4c29-8d60-1ab86165e48d","Type":"ContainerDied","Data":"775e4078b1b20cb3e9c11a220c2863b3feca8478f4eed8979e2f2913d0cbfa6b"}
Nov 02 20:13:14.789375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.789349    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:14.850387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.850358    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\") pod \"caec533d-5e2d-4c29-8d60-1ab86165e48d\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") "
Nov 02 20:13:14.850566 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.850426    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\" (UniqueName: \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\") pod \"caec533d-5e2d-4c29-8d60-1ab86165e48d\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") "
Nov 02 20:13:14.850619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.850577    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\") pod \"caec533d-5e2d-4c29-8d60-1ab86165e48d\" (UID: \"caec533d-5e2d-4c29-8d60-1ab86165e48d\") "
Nov 02 20:13:14.850968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.850938    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca" pod "caec533d-5e2d-4c29-8d60-1ab86165e48d" (UID: "caec533d-5e2d-4c29-8d60-1ab86165e48d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:14.851121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.851097    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:14.852645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.852607    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca" pod "caec533d-5e2d-4c29-8d60-1ab86165e48d" (UID: "caec533d-5e2d-4c29-8d60-1ab86165e48d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:14.853032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.853010    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx" pod "caec533d-5e2d-4c29-8d60-1ab86165e48d" (UID: "caec533d-5e2d-4c29-8d60-1ab86165e48d"). InnerVolumeSpecName "kube-api-access-5rsmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:14.952369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.952281    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsmx\" (UniqueName: \"kubernetes.io/projected/caec533d-5e2d-4c29-8d60-1ab86165e48d-kube-api-access-5rsmx\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:14.952369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:14.952312    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caec533d-5e2d-4c29-8d60-1ab86165e48d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:15.630938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:15.630908    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"
Nov 02 20:13:15.631110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:15.630949    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd" event={"ID":"caec533d-5e2d-4c29-8d60-1ab86165e48d","Type":"ContainerDied","Data":"8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779"}
Nov 02 20:13:15.631110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:15.630996    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8986700467c992f0b2bb2655b632aee30624b3bf9bbc142c4621a5774867e779"
Nov 02 20:13:16.094412 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:13:16.094329    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:13:16.634847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.634817    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:13:16.635512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.635487    5099 cpu_manager.go:414] "containerMap: removing container" podUID="caec533d-5e2d-4c29-8d60-1ab86165e48d" containerName="revoke-keys"
Nov 02 20:13:16.635512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.635510    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caec533d-5e2d-4c29-8d60-1ab86165e48d" containerName="revoke-keys"
Nov 02 20:13:16.635716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.635645    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="caec533d-5e2d-4c29-8d60-1ab86165e48d" containerName="revoke-keys"
Nov 02 20:13:16.638250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.638230    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.741418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.741387    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:13:16.775337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.775301    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvpz\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.775425    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.775861    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776038    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776103    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776144    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776207    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776244    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.776421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.776290    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878055    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878094    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878135    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878158    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878182    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878325    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvpz\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878388    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878413    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878486    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.878634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.878529    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.879109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.879083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.879357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.879340    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.879408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.879383    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.879491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.879477    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.881108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.881077    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.881207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.881189    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.933652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.933581    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.933652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.933641    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvpz\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.950677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.950653    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:16.998362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:16.998323    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:13:17.003165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.003137    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.058818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.058752    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:13:17.081602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081551    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.081800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081625    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.081800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081728    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.081928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081815    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.081928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081891    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.082018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.081946    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.082064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.082048    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.082115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.082102    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.082167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.082123    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jnf\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183365    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183434    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183482    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183526    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183584    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.183915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183610    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jnf\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183941    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.183999    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184023    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184067    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184386    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184394    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.184436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.184417    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.186122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.186093    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.186239 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.186224    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.237216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.237184    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.237423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.237401    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jnf\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.294426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.294393    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:13:17.297950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.297931    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.323544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.323513    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:17.353711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.353670    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:13:17.376949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.376909    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:13:17.386158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386129    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386198    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386335    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386509    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386563    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386639    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386705    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386742    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.386846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.386831    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2z6j\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.392294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.392256    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.392404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.392345    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.392404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.392388    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488461    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488504    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488585    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488605    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488623    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488662    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488680    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488717    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2z6j\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.488995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.488768    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.489630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.489317    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.489630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.489578    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.489885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.489842    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.490259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.490150    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.490259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.490228    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.491466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.491433    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.492146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.492126    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.543919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.543889    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.544381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.544359    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2z6j\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.610763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.610734    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:17.649948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.649910    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" event={"ID":"35c524ca-7874-48a5-82fc-7a5225208e08","Type":"ContainerStarted","Data":"1b74600168e074d10136f5e3a4fd78df7c8368eb2fc14439ae02cdef08fa6d55"}
Nov 02 20:13:17.650100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.649992    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" event={"ID":"35c524ca-7874-48a5-82fc-7a5225208e08","Type":"ContainerStarted","Data":"9588dea839494b84ed3dee6f51de47b4842dbfe5cfb9a2dd84628af1634ec922"}
Nov 02 20:13:17.720828 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:17.720766    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103abf05_c5fd_4b0a_9919_615b748b59f9.slice/crio-4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75 WatchSource:0}: Error finding container 4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75: Status 404 returned error can't find the container with id 4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75
Nov 02 20:13:17.724430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.724392    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.724517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.724480    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.724558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.724522    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:17.729501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.729474    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:13:17.774162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.774103    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podStartSLOduration=1.774082151 podStartE2EDuration="1.774082151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:17.773215123 +0000 UTC m=+1286.647486011" watchObservedRunningTime="2026-11-02 20:13:17.774082151 +0000 UTC m=+1286.648353044"
Nov 02 20:13:17.918814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.918759    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:13:17.925723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.923128    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:17.931478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.931448    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:13:17.935577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.935558    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:17.950776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.950744    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:17.981724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:17.981687    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.015923    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.015977    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.016008    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.016139    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.016188    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.016226    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.019452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.016328    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpwb\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.026086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.026048    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.026220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.026128    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.030777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.030034    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:13:18.034321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.033822    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:18.034321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.033921    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:18.034321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.033961    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:18.054130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.037411    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:13:18.127751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127710    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpwb\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.127913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127765    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.127913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127851    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.127913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127879    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127923    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127954    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.127978    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128005    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128037    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128068    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128171    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128212    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128496    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128553    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128555    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128584    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chqj\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.128862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128805    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.128975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.128894    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.129029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.129005    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.129084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.129065    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.129244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.129165    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.129545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.129382    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.129545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.129494    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.134523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.133403    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.134523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.133456    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.183042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.182981    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpwb\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.183198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.183119    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.221470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.221436    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:13:18.228154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.228127    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.230158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.230114    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.230371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.230351    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.230557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.230538    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chqj\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.230737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.230711    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.230887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.230871    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.231090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.231071    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.231304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.231276    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.231445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.231426    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.231910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.231878    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.232618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.232437    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.234022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.233989    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.235242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.235201    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.242808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.238609    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.242808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.238915    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.243484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.243459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.249196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.247620    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.250641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.250619    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:18.297023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.296953    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.319125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.318547    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:13:18.319461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.319429    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chqj\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.334851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.334621    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.334851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.334680    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335179    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335225    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335270    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335297    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335314    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335353    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfb4\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.335663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.335418    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.449098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.436942    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452429    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452501    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452588    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452636    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452662    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452706    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfb4\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452801    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.453489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.452910    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.456845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.454407    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.456845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.455108    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.456845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.455332    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.456845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.455926    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.456845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.456518    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.460247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.460188    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.467386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.467338    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.532223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.525891    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.629109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.558035    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:18.729305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.729171    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfb4\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.892203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.891617    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" event={"ID":"103abf05-c5fd-4b0a-9919-615b748b59f9","Type":"ContainerStarted","Data":"387da1f747206c76ead1ad0f6731089201bddc799f9f28465264c5d90fb105dd"}
Nov 02 20:13:18.892203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.891738    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" event={"ID":"103abf05-c5fd-4b0a-9919-615b748b59f9","Type":"ContainerStarted","Data":"4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75"}
Nov 02 20:13:18.895797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.895750    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" event={"ID":"20521a86-d32b-40cf-b237-c4d7e6b4e087","Type":"ContainerStarted","Data":"efd562d43958ae12ee0a6e77458b61b9328efa008273a040a9db372453843d2c"}
Nov 02 20:13:18.896014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.895993    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" event={"ID":"20521a86-d32b-40cf-b237-c4d7e6b4e087","Type":"ContainerStarted","Data":"5f837b9914660c1075ab8e1fc5ec136b946f848616e0565d9655657c6e1d0956"}
Nov 02 20:13:18.906836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.906406    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:18.977914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.977881    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:18.978317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:18.978266    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:19.054371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.054266    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:13:19.127022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.122735    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.127022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.122862    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.127022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.122909    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.127334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.127247    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podStartSLOduration=3.1272262570000002 podStartE2EDuration="3.127226257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.038282132 +0000 UTC m=+1287.912553021" watchObservedRunningTime="2026-11-02 20:13:19.127226257 +0000 UTC m=+1288.001497148"
Nov 02 20:13:19.137948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.137884    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"]
Nov 02 20:13:19.217959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.217891    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podStartSLOduration=2.21786807 podStartE2EDuration="2.21786807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.173296808 +0000 UTC m=+1288.047567699" watchObservedRunningTime="2026-11-02 20:13:19.21786807 +0000 UTC m=+1288.092138960"
Nov 02 20:13:19.223607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.223572    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53f9f72-lbxfj"]
Nov 02 20:13:19.337070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.336465    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f6a568-5155-4ba7-bd67-2e929d9006fb" path="/var/lib/kubelet/pods/50f6a568-5155-4ba7-bd67-2e929d9006fb/volumes"
Nov 02 20:13:19.337070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.337021    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:19.376014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.359105    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:19.376014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.359162    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:19.625878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.616017    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:19.625878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.623285    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:19.625878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.623357    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:19.886986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.875714    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.887303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.887273    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.887488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.887461    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:19.990987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.946008    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:13:19.990987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.959732    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:19.990987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:19.959836    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:20.026717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.026674    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:13:20.200806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.179343    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:20.221061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.221020    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:20.221297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.221278    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:20.284807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.281370    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" event={"ID":"8a2ad25a-302e-4c06-a539-1df4b92c5756","Type":"ContainerStarted","Data":"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"}
Nov 02 20:13:20.284807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.281439    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" event={"ID":"8a2ad25a-302e-4c06-a539-1df4b92c5756","Type":"ContainerStarted","Data":"c62f4939b7b0afbd2d4ab776a6ba1b8af536e9ad1844e0a8ad25e2c689b5be44"}
Nov 02 20:13:20.363818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.347158    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:20.363818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.347218    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:20.363818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.353350    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" event={"ID":"eba9eaf8-c2c4-4ad0-af03-418b98d3336c","Type":"ContainerStarted","Data":"c5e8286a2ddc8f85407c62024cbd00150713d049712616411cd8dea125eb0fc4"}
Nov 02 20:13:20.496861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.491241    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podStartSLOduration=3.4912193289999998 podStartE2EDuration="3.491219329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:20.462683995 +0000 UTC m=+1289.336954885" watchObservedRunningTime="2026-11-02 20:13:20.491219329 +0000 UTC m=+1289.365490220"
Nov 02 20:13:20.617993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.617950    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:20.618746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.618253    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:20.952271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.951127    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:20.952271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:20.951193    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:21.125815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.124890    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:21.529860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.464355    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:21.552807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.541323    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:21.577252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.577209    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:21.578033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.577996    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:21.614950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.614325    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:21.615221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.615186    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:21.642823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.634701    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" event={"ID":"eba9eaf8-c2c4-4ad0-af03-418b98d3336c","Type":"ContainerStarted","Data":"8fc56ed6755f1c70e880e1dfb647a9dd6f05f0e516da39967c3a510dfa18c910"}
Nov 02 20:13:21.675548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.637729    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:21.675982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.675917    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:21.676158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.675827    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:21.676317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.676288    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" event={"ID":"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad","Type":"ContainerStarted","Data":"4673aa0e92cee412664753b60969ae36515a80054813e33fa2ce34849fc6c09a"}
Nov 02 20:13:21.676443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.676424    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" event={"ID":"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad","Type":"ContainerStarted","Data":"0ba881965d0a54b50e5098fd6b04173a9ec05fb6b9d59207cbdb2a991090bee5"}
Nov 02 20:13:21.929435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.929399    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:21.930132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.930078    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podStartSLOduration=4.930060842 podStartE2EDuration="4.930060842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:21.85362079 +0000 UTC m=+1290.727891679" watchObservedRunningTime="2026-11-02 20:13:21.930060842 +0000 UTC m=+1290.804331732"
Nov 02 20:13:21.965004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.964964    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:21.965293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.965255    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:21.965502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.965478    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:21.967275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.967248    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:21.967430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.965636    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:21.967536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:21.967520    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:22.178217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.169904    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podStartSLOduration=4.169881949 podStartE2EDuration="4.169881949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:22.139373385 +0000 UTC m=+1291.013644309" watchObservedRunningTime="2026-11-02 20:13:22.169881949 +0000 UTC m=+1291.044152834"
Nov 02 20:13:22.240325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.240229    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:22.341575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.341541    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:22.341868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.341837    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:22.414324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.414287    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:22.414672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.414636    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:22.547886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.545551    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"]
Nov 02 20:13:22.581807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.555607    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.606894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.577276    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:22.645810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.611328    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:22.645810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.611380    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:22.645810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.632046    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:22.645810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.644933    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.660107    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.660270    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.661054    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.661130    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.679424    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.679481    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.719406    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.719521    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.719805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.719613    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgt2\" (UniqueName: \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.840860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.832162    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.840860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.832275    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.840860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.832400    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgt2\" (UniqueName: \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.864810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.857652    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.883974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.883925    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:22.886429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.886404    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:22.929599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.929558    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:22.930206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.930180    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:22.934805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.931241    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:22.960266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.940636    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgt2\" (UniqueName: \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:22.994981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.994940    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:23.020805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:22.997473    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:23.020805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.016442    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:23.020805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.016497    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:23.075366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.075327    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"]
Nov 02 20:13:23.199851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.178120    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:23.256836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.246361    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:23.256836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.249192    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:13:23.382814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.373830    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:23.382814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.373897    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:23.382814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.377281    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:23.382814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.377335    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:23.382814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.379237    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f22ad2a-57mgc\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:23.598902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.532625    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:23.598902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.581365    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Nov 02 20:13:23.831810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.650186    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:23.890975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.678484    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:23.891440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.699041    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:23.900807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.891591    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:24.032504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.801912    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:24.032504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:23.941895    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:24.064991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.036548    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:13:24.064991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.057750    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:24.151892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.148852    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.151892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.148906    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:24.151892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.148983    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.151892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.149003    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:24.178143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.178105    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:24.243951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.178713    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.244232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.244184    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:24.292618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.292569    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:24.304755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.304719    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.481412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.481368    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:24.481689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.326905    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.481845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.481817    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:24.609818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.607419    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.609818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.607482    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:24.639813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.635255    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:24.640180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:24.640145    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:25.029819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.016750    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.043861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.030119    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:25.045742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.045692    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:25.068713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.068671    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:25.117671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.104205    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:13:25.186933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.186884    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:25.217018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.216977    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.247409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.247364    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:25.247901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.239355    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:13:25.262900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.262859    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:25.313335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.287285    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.313335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.287362    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:25.377276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.324497    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:25.679478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.679429    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.813053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.813000    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:25.904072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.904032    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.904366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.904332    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:25.985879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.979398    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:25.993811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:25.989889    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:26.027177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.010923    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.027177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.011011    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:26.059976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.059916    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:26.069950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.069904    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:26.211806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.109845    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:26.215807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.212632    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:26.441010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.353918    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:13:26.441432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.353971    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:13:26.441569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.354068    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.444851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.381136    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.462836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.445008    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:26.495808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.441690    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.438183    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.568652    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:13:26.569012    5099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.306s"
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.569684    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"]
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.569735    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a83434b-hq4cq"]
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.570221    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.570457    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:26.648949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.648905    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:26.649146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.648978    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:26.991817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.982627    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.030214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.030170    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:27.030464 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:26.989260    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.030585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.030563    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:27.061570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.061182    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:27.090001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.089966    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"]
Nov 02 20:13:27.119095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.119054    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:27.120804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.120763    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:27.130273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.128844    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"]
Nov 02 20:13:27.211851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.211674    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:27.212948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.212909    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:13:27.261814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.249111    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6317f9-7498v"]
Nov 02 20:13:27.279665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.278267    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.279665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.278329    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:27.346724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.346585    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.346724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.346648    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:27.462802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.454959    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:27.462802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.455358    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:27.531878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.491073    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cade08b-806e-4545-99a8-188f8c78d0ba" path="/var/lib/kubelet/pods/7cade08b-806e-4545-99a8-188f8c78d0ba/volumes"
Nov 02 20:13:27.606117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.606078    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.606364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.606334    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:27.651166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.638473    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4eda95-3565-4af4-8d05-695b586a1fac" path="/var/lib/kubelet/pods/7d4eda95-3565-4af4-8d05-695b586a1fac/volumes"
Nov 02 20:13:27.651166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.639445    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.651166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.639498    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:27.673909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.657904    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc" event={"ID":"f2666e00-41fa-44df-a950-a264d69448f1","Type":"ContainerStarted","Data":"5253f2bb2e011a80eb86131491bc134d2e51b2b44abfc5f0b1812b622b40b221"}
Nov 02 20:13:27.973527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.973480    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.973968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.973917    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:27.975616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.975578    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 02 20:13:27.975812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:27.975765    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 02 20:13:28.089932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.088922    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:13:28.216799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.216727    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:28.217768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.217071    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:28.249107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.249000    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:28.255606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.255570    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:28.255886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.255858    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:28.334617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.334582    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:28.334727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.334638    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:28.456981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.456928    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:28.459066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.457647    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:28.658847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.656122    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:28.658847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.656188    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:28.658847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.656272    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:28.658847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.656293    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:28.706807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.705452    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc" event={"ID":"f2666e00-41fa-44df-a950-a264d69448f1","Type":"ContainerStarted","Data":"0a2b2385f6a76fabf5fbb9831c38c3b096fc17257f958a75eabdb9965747c7df"}
Nov 02 20:13:28.919270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.919173    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:28.919517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:28.919487    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:29.231886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.178411    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:29.231886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.181700    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:13:29.231886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.224595    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:29.231886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.224702    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:29.231886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.225229    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:13:29.399830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.285351    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:29.399830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.285421    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:29.482867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.478888    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:29.494216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.480581    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:29.605897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.566007    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 02 20:13:29.605897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.566075    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 02 20:13:29.605897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.590189    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:29.605897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.590244    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:29.625384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.624373    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:29.625384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.624432    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:29.642517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.636463    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc" podStartSLOduration=7.636441422 podStartE2EDuration="7.636441422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:28.841404558 +0000 UTC m=+1297.715675445" watchObservedRunningTime="2026-11-02 20:13:29.636441422 +0000 UTC m=+1298.510712313"
Nov 02 20:13:29.772414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.772152    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:29.801891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.787829    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:29.932809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:29.812545    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:13:30.159818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.114479    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:30.159818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.114552    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:30.195811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.178745    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:30.236726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.236682    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:30.325710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.239325    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:30.325710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.272883    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:30.325710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.272953    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:30.325710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.317037    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:30.328184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.328140    5099 generic.go:356] "Generic (PLEG): container finished" podID="f2666e00-41fa-44df-a950-a264d69448f1" containerID="0a2b2385f6a76fabf5fbb9831c38c3b096fc17257f958a75eabdb9965747c7df" exitCode=0
Nov 02 20:13:30.346878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.328502    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc" event={"ID":"f2666e00-41fa-44df-a950-a264d69448f1","Type":"ContainerDied","Data":"0a2b2385f6a76fabf5fbb9831c38c3b096fc17257f958a75eabdb9965747c7df"}
Nov 02 20:13:30.514033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.513924    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:30.630638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.536141    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:30.640808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.611967    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:30.640808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.636389    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:30.640808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.628971    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 20:13:30.640808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.636459    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 20:13:30.919922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.919887    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:30.920480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:30.920288    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:31.152086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.152037    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:31.153893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.153870    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:31.223159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.223065    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:13:31.223804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.223763    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:31.226451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.226423    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Nov 02 20:13:31.242125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.239530    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:31.255237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.255201    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:31.255503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.255456    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:31.368810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.359575    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:13:31.771850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.767011    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:13:31.959278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:31.959241    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:32.007822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.001682    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:32.007822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.001744    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:32.007822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.002179    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:32.007822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.002220    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:32.042941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.017213    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:13:32.042941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.017317    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:32.042941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.017349    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:13:32.096367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.090569    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:32.096367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.090613    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:13:32.096367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.091299    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.216251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.216217    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304306    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304399    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304427    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304453    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304484    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304659    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304811    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkw6v\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.304885    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.294920    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.305504    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 20:13:32.305875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.295123    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:32.335560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.335529    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:32.442692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.442662    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:32.444347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.407139    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.444643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444615    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.444751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444658    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.444900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444777    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.444900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444840    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.444900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444888    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.445118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.444936    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.445200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.445144    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.445351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.445321    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkw6v\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.446535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.446499    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:32.450431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.449403    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.450431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.449629    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.450431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.450069    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:32.451517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.451479    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.456906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.456864    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:13:32.483117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.483079    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.496879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.496844    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:13:32.506850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.503170    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.506850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.503630    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.510520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.510362    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.535658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.535125    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkw6v\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.577728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.573055    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:32.577728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.573109    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:32.577728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.575410    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.696163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.689041    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:32.810809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.810706    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:32.919518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.919479    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 20:13:32.919691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.919543    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 20:13:32.986807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.986702    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\" (UniqueName: \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\") pod \"f2666e00-41fa-44df-a950-a264d69448f1\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") "
Nov 02 20:13:32.987002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.986859    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\") pod \"f2666e00-41fa-44df-a950-a264d69448f1\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") "
Nov 02 20:13:32.987002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.986962    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\") pod \"f2666e00-41fa-44df-a950-a264d69448f1\" (UID: \"f2666e00-41fa-44df-a950-a264d69448f1\") "
Nov 02 20:13:32.996366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.991320    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca" pod "f2666e00-41fa-44df-a950-a264d69448f1" (UID: "f2666e00-41fa-44df-a950-a264d69448f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:32.996366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.994024    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2" pod "f2666e00-41fa-44df-a950-a264d69448f1" (UID: "f2666e00-41fa-44df-a950-a264d69448f1"). InnerVolumeSpecName "kube-api-access-ncgt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:32.996366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:32.994342    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca" pod "f2666e00-41fa-44df-a950-a264d69448f1" (UID: "f2666e00-41fa-44df-a950-a264d69448f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:33.022039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.018668    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"
Nov 02 20:13:33.022039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.021619    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc" event={"ID":"f2666e00-41fa-44df-a950-a264d69448f1","Type":"ContainerDied","Data":"5253f2bb2e011a80eb86131491bc134d2e51b2b44abfc5f0b1812b622b40b221"}
Nov 02 20:13:33.022039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.021711    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5253f2bb2e011a80eb86131491bc134d2e51b2b44abfc5f0b1812b622b40b221"
Nov 02 20:13:33.103063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.095173    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:33.103063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.095210    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgt2\" (UniqueName: \"kubernetes.io/projected/f2666e00-41fa-44df-a950-a264d69448f1-kube-api-access-ncgt2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:33.103063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.095227    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2666e00-41fa-44df-a950-a264d69448f1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:33.326663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.325472    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:33.358874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.358830    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:33.359124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.359103    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:33.359304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.359281    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:33.362670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.362635    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:13:33.395438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.395287    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:13:33.397019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.396993    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f2666e00-41fa-44df-a950-a264d69448f1" containerName="revoke-keys"
Nov 02 20:13:33.397137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.397020    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2666e00-41fa-44df-a950-a264d69448f1" containerName="revoke-keys"
Nov 02 20:13:33.397452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.397432    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2666e00-41fa-44df-a950-a264d69448f1" containerName="revoke-keys"
Nov 02 20:13:33.420865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.418735    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:13:33.420865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.418862    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.528973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.528834    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.529145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.528969    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.529145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529128    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529305    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529342    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlt6\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529402    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529463    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529556    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.531457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.529650    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.559955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.558739    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:33.559955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.559439    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 20:13:33.559955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.559480    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 20:13:33.605800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.605696    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:33.636563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636045    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.636563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636131    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.636563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636332    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.636563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636509    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.638181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636719    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.638181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636749    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlt6\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.638181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.636822    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.640983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.638879    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.640983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.638974    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.640983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.639490    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.640983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.639727    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.640983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.640403    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.641905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.641880    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.663812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.642123    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.705813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.696692    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.708748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.707151    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.737476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.737406    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.738350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.738326    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlt6\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.863440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.863160    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:33.920180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:33.918647    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:34.060725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.060687    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" event={"ID":"d4602a17-a939-4de6-9ab6-c3fc244164e0","Type":"ContainerStarted","Data":"89222753b5e8a37eec5a8cb65aad9add555ec0328e49def9612301ae0cf17669"}
Nov 02 20:13:34.063002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.062954    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" event={"ID":"d4602a17-a939-4de6-9ab6-c3fc244164e0","Type":"ContainerStarted","Data":"7dcfb37554432af6940bad80e78be5db4d4f221eecccc8fbea8cfdd1b2af5665"}
Nov 02 20:13:34.063404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.063386    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:13:34.206554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.206449    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podStartSLOduration=3.20643035 podStartE2EDuration="3.20643035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:34.197336334 +0000 UTC m=+1303.071607227" watchObservedRunningTime="2026-11-02 20:13:34.20643035 +0000 UTC m=+1303.080701239"
Nov 02 20:13:34.351162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.351123    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:13:34.403979 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.403942    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:34.404098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.404054    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:34.404164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.404099    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:13:34.542068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.533622    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:34.542068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.538636    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:13:34.630082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.630043    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:34.635097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.635060    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:13:34.697970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.697219    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:34.700960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.700595    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:34.700960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:34.700661    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:35.083864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.083820    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" event={"ID":"3b23f814-fed9-40fa-8551-ce1f6715ffc8","Type":"ContainerStarted","Data":"f19b6447e0a4e5a05b7263063015ea8c9b82f386e4b5a313c0144e806aa587d9"}
Nov 02 20:13:35.084069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.083886    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" event={"ID":"3b23f814-fed9-40fa-8551-ce1f6715ffc8","Type":"ContainerStarted","Data":"383b8d25dbd151640e9d4f710164485c7b82b3da4a9f00add0b9852d72bd10e8"}
Nov 02 20:13:35.198834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.198771    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podStartSLOduration=2.198758976 podStartE2EDuration="2.198758976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:35.195528516 +0000 UTC m=+1304.069799406" watchObservedRunningTime="2026-11-02 20:13:35.198758976 +0000 UTC m=+1304.073029864"
Nov 02 20:13:35.712816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.712267    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:35.712816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.712346    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:35.877281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.877185    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:35.887074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.886939    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:35.887074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:35.887001    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:36.306645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.306602    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"]
Nov 02 20:13:36.313328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.313300    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.345000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.344960    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"]
Nov 02 20:13:36.414769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.414721    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:36.417241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.415689    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4sd\" (UniqueName: \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.417575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.417548    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.417846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.417191    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:36.417947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.417814    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.519027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.518986    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.519222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.519124    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.519288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.519241    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4sd\" (UniqueName: \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.520279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.520239    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.522630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.522605    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.582532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.582451    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4sd\" (UniqueName: \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\") pod \"maas-api-revoke-sub-6acd604b-gqkph\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.628586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.628384    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:36.700835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.700776    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:36.701016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.700855    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:36.831893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.831373    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:13:36.864858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.858932    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:13:36.884808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.865185    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:36.885827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.875137    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:36.887065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.887031    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.958180    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"openshift-service-ca.crt\""
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.958973    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"default-dockercfg-sqlwk\""
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.959079    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"distinct-2-w1-accdf5-kserve-self-signed-certs\""
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.959171    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"kube-root-ca.crt\""
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.960520    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b82s\" (UniqueName: \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.960605    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.960633    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.960698    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.960865    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.056121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:36.961037    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078450    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078609    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078699    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b82s\" (UniqueName: \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078757    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078804    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.078862    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.079940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.079521    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.167173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.163142    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.274825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.267295    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.563278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.563198    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b82s\" (UniqueName: \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.690655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.600327    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.690655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.690563    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\") pod \"distinct-2-w1-accdf5-kserve-6d5856b957-clzvw\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.693638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.693390    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:37.717591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.717556    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:37.717756 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.717616    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:37.814502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.814411    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"]
Nov 02 20:13:37.902452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.899875    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:37.902452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:37.899940    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:38.232476 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:38.232438    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e6dc5a8_18b5_4715_8ff6_584df80cded0.slice/crio-e8ab9d164810cd6dcd413d311ce872d6d549e84f13f85e2b67a5f682eb5d7365 WatchSource:0}: Error finding container e8ab9d164810cd6dcd413d311ce872d6d549e84f13f85e2b67a5f682eb5d7365: Status 404 returned error can't find the container with id e8ab9d164810cd6dcd413d311ce872d6d549e84f13f85e2b67a5f682eb5d7365
Nov 02 20:13:38.232476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.232575    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:13:38.723861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.723747    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:38.723861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.723827    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:38.817059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.817026    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" event={"ID":"9e6dc5a8-18b5-4715-8ff6-584df80cded0","Type":"ContainerStarted","Data":"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"}
Nov 02 20:13:38.817226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.817070    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" event={"ID":"9e6dc5a8-18b5-4715-8ff6-584df80cded0","Type":"ContainerStarted","Data":"e8ab9d164810cd6dcd413d311ce872d6d549e84f13f85e2b67a5f682eb5d7365"}
Nov 02 20:13:38.820125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.820094    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph" event={"ID":"f36b0c96-08b8-4bd6-a136-439bdbc8ba80","Type":"ContainerStarted","Data":"e7ffd72cda64524e9c82b1d0ca7c7029c9ec0b448b056ddb25acc5e2b08eceab"}
Nov 02 20:13:38.820302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.820266    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph" event={"ID":"f36b0c96-08b8-4bd6-a136-439bdbc8ba80","Type":"ContainerStarted","Data":"d47367c92c1b07b72ba5074153ef05f165ed28369d072c81c825c56b0cadf4be"}
Nov 02 20:13:38.878606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.874145    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:38.878606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.874207    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:38.980882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:38.979890    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podStartSLOduration=2.979863865 podStartE2EDuration="2.979863865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:38.945165803 +0000 UTC m=+1307.819436692" watchObservedRunningTime="2026-11-02 20:13:38.979863865 +0000 UTC m=+1307.854134754"
Nov 02 20:13:39.066054 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.065971    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph" podStartSLOduration=3.065936828 podStartE2EDuration="3.065936828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:39.064707097 +0000 UTC m=+1307.938977990" watchObservedRunningTime="2026-11-02 20:13:39.065936828 +0000 UTC m=+1307.940207717"
Nov 02 20:13:39.697389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.697347    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:39.697661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.697633    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:39.698190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.698168    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:39.830249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.830219    5099 generic.go:356] "Generic (PLEG): container finished" podID="f36b0c96-08b8-4bd6-a136-439bdbc8ba80" containerID="e7ffd72cda64524e9c82b1d0ca7c7029c9ec0b448b056ddb25acc5e2b08eceab" exitCode=0
Nov 02 20:13:39.830406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.830313    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph" event={"ID":"f36b0c96-08b8-4bd6-a136-439bdbc8ba80","Type":"ContainerDied","Data":"e7ffd72cda64524e9c82b1d0ca7c7029c9ec0b448b056ddb25acc5e2b08eceab"}
Nov 02 20:13:39.874805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.874538    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:39.874805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:39.874599    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:40.087926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.087832    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"]
Nov 02 20:13:40.104819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.104368    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c414e206-ssl7t"]
Nov 02 20:13:40.705859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.702032    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:40.705859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.702108    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:40.918498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.895979    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:40.918498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:40.896040    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:41.656318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:41.656277    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce" path="/var/lib/kubelet/pods/3c1be8ff-bfbb-40e6-8c2d-2b3cf26096ce/volumes"
Nov 02 20:13:41.710395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:41.710344    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:41.710684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:41.710651    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:41.944996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:41.944903    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:42.009035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.008986    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:42.233865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.232045    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:42.552738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.552659    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:42.651807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.645153    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:42.651807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.646832    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:42.688807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.677576    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\" (UniqueName: \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\") pod \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") "
Nov 02 20:13:42.688807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.677669    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\") pod \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") "
Nov 02 20:13:42.688807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.677793    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\") pod \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\" (UID: \"f36b0c96-08b8-4bd6-a136-439bdbc8ba80\") "
Nov 02 20:13:42.688807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.679173    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca" pod "f36b0c96-08b8-4bd6-a136-439bdbc8ba80" (UID: "f36b0c96-08b8-4bd6-a136-439bdbc8ba80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:42.688807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.687874    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:42.704906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.704856    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 02 20:13:42.705187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.705140    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 02 20:13:42.731175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.731139    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd" pod "f36b0c96-08b8-4bd6-a136-439bdbc8ba80" (UID: "f36b0c96-08b8-4bd6-a136-439bdbc8ba80"). InnerVolumeSpecName "kube-api-access-ks4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:42.809076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.808983    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca" pod "f36b0c96-08b8-4bd6-a136-439bdbc8ba80" (UID: "f36b0c96-08b8-4bd6-a136-439bdbc8ba80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:42.817863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.811180    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4sd\" (UniqueName: \"kubernetes.io/projected/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-kube-api-access-ks4sd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:42.818117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.818095    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:42.895262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.893944    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:42.895262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.894010    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:42.928590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.928385    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36b0c96-08b8-4bd6-a136-439bdbc8ba80-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:42.946863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:42.932903    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:13:43.127975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.127921    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:43.464317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.464221    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph" event={"ID":"f36b0c96-08b8-4bd6-a136-439bdbc8ba80","Type":"ContainerDied","Data":"d47367c92c1b07b72ba5074153ef05f165ed28369d072c81c825c56b0cadf4be"}
Nov 02 20:13:43.464564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.464542    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47367c92c1b07b72ba5074153ef05f165ed28369d072c81c825c56b0cadf4be"
Nov 02 20:13:43.464738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.464724    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"
Nov 02 20:13:43.535128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.533591    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Nov 02 20:13:43.649810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.649114    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:43.649810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.649486    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:43.661807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.650545    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:13:43.661807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.650613    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:43.706818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.696896    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:43.706818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.697355    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:43.803058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.802957    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:43.941853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.930325    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 20:13:43.941853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.930381    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 20:13:43.961822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.942845    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:13:43.961822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:43.943221    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:13:44.107643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.107554    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:13:44.132913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.132865    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:44.138812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.133510    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:44.335413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.279956    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:13:44.677827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.670895    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:13:44.749917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.749861    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:13:44.841882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.800999    5099 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:13:44.841882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.801063    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:44.947168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:44.946962    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:45.133905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.133856    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:13:45.331853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.331728    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:13:45.332687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.332661    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:45.349365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.349332    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:13:45.445079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.438876    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:45.451802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.445406    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:13:45.475097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.475069    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:45.511296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.511092    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:13:45.528527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.528498    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:13:45.531336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.531315    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:13:45.532334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.532316    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:13:45.532479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.532464    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"]
Nov 02 20:13:45.541847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.541824    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a1063-jgt8m"]
Nov 02 20:13:45.705726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.705690    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:45.706463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.706438    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:45.707340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:45.707308    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:13:46.159488 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:13:46.159448    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:13:47.260111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:47.260082    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf30ab4f-ad0d-4381-840a-1130d2947d59" path="/var/lib/kubelet/pods/cf30ab4f-ad0d-4381-840a-1130d2947d59/volumes"
Nov 02 20:13:47.696631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:47.696597    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:47.709097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:47.709070    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:48.264132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:48.264094    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:13:50.046573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.046538    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"]
Nov 02 20:13:50.093586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.093546    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fc7448-vjljs"]
Nov 02 20:13:50.266526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.266491    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"]
Nov 02 20:13:50.267577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.267555    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f36b0c96-08b8-4bd6-a136-439bdbc8ba80" containerName="revoke-keys"
Nov 02 20:13:50.267622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.267579    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36b0c96-08b8-4bd6-a136-439bdbc8ba80" containerName="revoke-keys"
Nov 02 20:13:50.267762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.267750    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36b0c96-08b8-4bd6-a136-439bdbc8ba80" containerName="revoke-keys"
Nov 02 20:13:50.272242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.272219    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.320410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.320334    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"]
Nov 02 20:13:50.321825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.321798    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:50.322171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.322149    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:50.379380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.378940    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.379380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.379195    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbprm\" (UniqueName: \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.379380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.379324    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.481221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.481178    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.481423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.481406    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbprm\" (UniqueName: \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.481509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.481495    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.482191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.482165    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.483801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.483750    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.535378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.535345    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbprm\" (UniqueName: \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\") pod \"maas-api-revoke-sub-9795afc2-hfzbf\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.587992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.587909    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:50.960572 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:50.960527    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bcd50b_ea1f_428f_8a1c_9975d613919d.slice/crio-cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236 WatchSource:0}: Error finding container cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236: Status 404 returned error can't find the container with id cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236
Nov 02 20:13:50.962183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:50.962151    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"]
Nov 02 20:13:51.048575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.048542    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"]
Nov 02 20:13:51.054016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.053984    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf439863-n46dc"]
Nov 02 20:13:51.276958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.276916    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c91fc7-1fcd-440b-8e75-535a3991cedb" path="/var/lib/kubelet/pods/d5c91fc7-1fcd-440b-8e75-535a3991cedb/volumes"
Nov 02 20:13:51.278083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.277761    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17becb9-5fdc-481f-9fd2-c0fb61a12722" path="/var/lib/kubelet/pods/e17becb9-5fdc-481f-9fd2-c0fb61a12722/volumes"
Nov 02 20:13:51.278280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.278252    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf" event={"ID":"52bcd50b-ea1f-428f-8a1c-9975d613919d","Type":"ContainerStarted","Data":"cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236"}
Nov 02 20:13:51.386837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.386768    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf" podStartSLOduration=1.386755427 podStartE2EDuration="1.386755427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:51.385388526 +0000 UTC m=+1320.259659416" watchObservedRunningTime="2026-11-02 20:13:51.386755427 +0000 UTC m=+1320.261026314"
Nov 02 20:13:51.943990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.943957    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"]
Nov 02 20:13:51.948910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:51.948889    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.000404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.000371    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"]
Nov 02 20:13:52.003249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.003222    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.003401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.003286    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.003463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.003394    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78kk\" (UniqueName: \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.052659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.052558    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"]
Nov 02 20:13:52.057445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.057413    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab0e09a-5jr87"]
Nov 02 20:13:52.105060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.105024    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.105229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.105094    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.105229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.105159    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78kk\" (UniqueName: \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.105709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.105672    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.107573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.107550    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.159089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.159061    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78kk\" (UniqueName: \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\") pod \"maas-api-revoke-sub-b506056e-7hw2z\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.262870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.262839    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:52.287184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.287151    5099 generic.go:356] "Generic (PLEG): container finished" podID="52bcd50b-ea1f-428f-8a1c-9975d613919d" containerID="013a17dd1dbd2044902e7da93e17e31089f88b9e25654ad89640e9c3f689ad65" exitCode=0
Nov 02 20:13:52.287317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.287230    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf" event={"ID":"52bcd50b-ea1f-428f-8a1c-9975d613919d","Type":"ContainerDied","Data":"013a17dd1dbd2044902e7da93e17e31089f88b9e25654ad89640e9c3f689ad65"}
Nov 02 20:13:52.642567 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:52.642536    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67631a35_4b95_45bf_abcb_a9aec002d6ff.slice/crio-eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d WatchSource:0}: Error finding container eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d: Status 404 returned error can't find the container with id eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d
Nov 02 20:13:52.643864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:52.643835    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"]
Nov 02 20:13:53.258911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.258834    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1567b2e-c7e2-43ad-ad61-f183144fdacd" path="/var/lib/kubelet/pods/d1567b2e-c7e2-43ad-ad61-f183144fdacd/volumes"
Nov 02 20:13:53.297354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.297316    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z" event={"ID":"67631a35-4b95-45bf-abcb-a9aec002d6ff","Type":"ContainerStarted","Data":"ee86786f59ba66bdb490f1ed03336a38af9bef1d02db963e50a18829841c71ce"}
Nov 02 20:13:53.297520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.297370    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z" event={"ID":"67631a35-4b95-45bf-abcb-a9aec002d6ff","Type":"ContainerStarted","Data":"eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d"}
Nov 02 20:13:53.409419 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.409366    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z" podStartSLOduration=2.409350206 podStartE2EDuration="2.409350206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:53.403425296 +0000 UTC m=+1322.277696185" watchObservedRunningTime="2026-11-02 20:13:53.409350206 +0000 UTC m=+1322.283621094"
Nov 02 20:13:53.470591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.470564    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:53.525550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.525464    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\" (UniqueName: \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\") pod \"52bcd50b-ea1f-428f-8a1c-9975d613919d\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") "
Nov 02 20:13:53.525696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.525554    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\") pod \"52bcd50b-ea1f-428f-8a1c-9975d613919d\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") "
Nov 02 20:13:53.525696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.525667    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\") pod \"52bcd50b-ea1f-428f-8a1c-9975d613919d\" (UID: \"52bcd50b-ea1f-428f-8a1c-9975d613919d\") "
Nov 02 20:13:53.525998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.525960    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca" pod "52bcd50b-ea1f-428f-8a1c-9975d613919d" (UID: "52bcd50b-ea1f-428f-8a1c-9975d613919d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:53.526242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.526226    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:53.527813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.527772    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca" pod "52bcd50b-ea1f-428f-8a1c-9975d613919d" (UID: "52bcd50b-ea1f-428f-8a1c-9975d613919d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:53.527813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.527800    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm" pod "52bcd50b-ea1f-428f-8a1c-9975d613919d" (UID: "52bcd50b-ea1f-428f-8a1c-9975d613919d"). InnerVolumeSpecName "kube-api-access-dbprm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:53.627598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.627556    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52bcd50b-ea1f-428f-8a1c-9975d613919d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:53.627598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:53.627597    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbprm\" (UniqueName: \"kubernetes.io/projected/52bcd50b-ea1f-428f-8a1c-9975d613919d-kube-api-access-dbprm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:54.306454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:54.306424    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"
Nov 02 20:13:54.306953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:54.306463    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf" event={"ID":"52bcd50b-ea1f-428f-8a1c-9975d613919d","Type":"ContainerDied","Data":"cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236"}
Nov 02 20:13:54.306953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:54.306512    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cccff5c2ace4be25c272b1d24621ab023e0ee1815212fcb10a2799ba0d19e236"
Nov 02 20:13:54.308048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:54.308022    5099 generic.go:356] "Generic (PLEG): container finished" podID="67631a35-4b95-45bf-abcb-a9aec002d6ff" containerID="ee86786f59ba66bdb490f1ed03336a38af9bef1d02db963e50a18829841c71ce" exitCode=0
Nov 02 20:13:54.308163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:54.308095    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z" event={"ID":"67631a35-4b95-45bf-abcb-a9aec002d6ff","Type":"ContainerDied","Data":"ee86786f59ba66bdb490f1ed03336a38af9bef1d02db963e50a18829841c71ce"}
Nov 02 20:13:55.504152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.504126    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:55.553406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.553369    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\" (UniqueName: \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\") pod \"67631a35-4b95-45bf-abcb-a9aec002d6ff\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") "
Nov 02 20:13:55.553406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.553411    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\") pod \"67631a35-4b95-45bf-abcb-a9aec002d6ff\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") "
Nov 02 20:13:55.553636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.553552    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\") pod \"67631a35-4b95-45bf-abcb-a9aec002d6ff\" (UID: \"67631a35-4b95-45bf-abcb-a9aec002d6ff\") "
Nov 02 20:13:55.553991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.553961    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca" pod "67631a35-4b95-45bf-abcb-a9aec002d6ff" (UID: "67631a35-4b95-45bf-abcb-a9aec002d6ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:13:55.554245 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.554227    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:55.555544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.555520    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca" pod "67631a35-4b95-45bf-abcb-a9aec002d6ff" (UID: "67631a35-4b95-45bf-abcb-a9aec002d6ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:13:55.555632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.555545    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk" pod "67631a35-4b95-45bf-abcb-a9aec002d6ff" (UID: "67631a35-4b95-45bf-abcb-a9aec002d6ff"). InnerVolumeSpecName "kube-api-access-c78kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:13:55.656096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.656050    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78kk\" (UniqueName: \"kubernetes.io/projected/67631a35-4b95-45bf-abcb-a9aec002d6ff-kube-api-access-c78kk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:55.656096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:55.656092    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67631a35-4b95-45bf-abcb-a9aec002d6ff-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:13:56.326708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:56.326670    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z" event={"ID":"67631a35-4b95-45bf-abcb-a9aec002d6ff","Type":"ContainerDied","Data":"eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d"}
Nov 02 20:13:56.326909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:56.326728    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eedee2b324c7ebaef3922a36882ee9cd4dc343bd73977701d018f9755eb34b5d"
Nov 02 20:13:56.326909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:56.326685    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"
Nov 02 20:13:59.039301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.039262    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"]
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040363    5099 cpu_manager.go:414] "containerMap: removing container" podUID="52bcd50b-ea1f-428f-8a1c-9975d613919d" containerName="revoke-keys"
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040384    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bcd50b-ea1f-428f-8a1c-9975d613919d" containerName="revoke-keys"
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040448    5099 cpu_manager.go:414] "containerMap: removing container" podUID="67631a35-4b95-45bf-abcb-a9aec002d6ff" containerName="revoke-keys"
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040457    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67631a35-4b95-45bf-abcb-a9aec002d6ff" containerName="revoke-keys"
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040636    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="67631a35-4b95-45bf-abcb-a9aec002d6ff" containerName="revoke-keys"
Nov 02 20:13:59.041161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.040662    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bcd50b-ea1f-428f-8a1c-9975d613919d" containerName="revoke-keys"
Nov 02 20:13:59.045276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.045236    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.093944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.093914    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"]
Nov 02 20:13:59.106771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.106735    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.107048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.107019    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzwf\" (UniqueName: \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.107143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.107102    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.135820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.135688    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:13:59.135820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.135749    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:13:59.208659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.208620    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.208976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.208821    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzwf\" (UniqueName: \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.208976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.208874    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.209546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.209524    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.211048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.211026    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.264769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.264733    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzwf\" (UniqueName: \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\") pod \"maas-api-revoke-sub-944488f9-l5gb5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.360689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.360654    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:13:59.734638 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:13:59.734603    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e40349_5135_47f6_816f_bce368e018a5.slice/crio-ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6 WatchSource:0}: Error finding container ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6: Status 404 returned error can't find the container with id ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6
Nov 02 20:13:59.737756 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:13:59.737722    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"]
Nov 02 20:14:00.410299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:00.410259    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5" event={"ID":"e6e40349-5135-47f6-816f-bce368e018a5","Type":"ContainerStarted","Data":"b899914bc52bec6f8cf864140b607366e0823c8e7fb295927f9d82c13bf40110"}
Nov 02 20:14:00.410718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:00.410321    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5" event={"ID":"e6e40349-5135-47f6-816f-bce368e018a5","Type":"ContainerStarted","Data":"ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6"}
Nov 02 20:14:00.520956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:00.520888    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5" podStartSLOduration=2.520873525 podStartE2EDuration="2.520873525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:00.516246928 +0000 UTC m=+1329.390517797" watchObservedRunningTime="2026-11-02 20:14:00.520873525 +0000 UTC m=+1329.395144446"
Nov 02 20:14:01.420281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:01.420244    5099 generic.go:356] "Generic (PLEG): container finished" podID="e6e40349-5135-47f6-816f-bce368e018a5" containerID="b899914bc52bec6f8cf864140b607366e0823c8e7fb295927f9d82c13bf40110" exitCode=0
Nov 02 20:14:01.420680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:01.420290    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5" event={"ID":"e6e40349-5135-47f6-816f-bce368e018a5","Type":"ContainerDied","Data":"b899914bc52bec6f8cf864140b607366e0823c8e7fb295927f9d82c13bf40110"}
Nov 02 20:14:02.044768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.044729    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"]
Nov 02 20:14:02.049345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.049315    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cb53246-j6f9s"]
Nov 02 20:14:02.604022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.603993    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:14:02.629535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.629501    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:14:02.631027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.630497    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e6e40349-5135-47f6-816f-bce368e018a5" containerName="revoke-keys"
Nov 02 20:14:02.631027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.630524    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e40349-5135-47f6-816f-bce368e018a5" containerName="revoke-keys"
Nov 02 20:14:02.631027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.630684    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e40349-5135-47f6-816f-bce368e018a5" containerName="revoke-keys"
Nov 02 20:14:02.634436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.634419    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.659561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.659534    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\") pod \"e6e40349-5135-47f6-816f-bce368e018a5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") "
Nov 02 20:14:02.659702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.659606    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\") pod \"e6e40349-5135-47f6-816f-bce368e018a5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") "
Nov 02 20:14:02.659842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.659827    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\" (UniqueName: \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\") pod \"e6e40349-5135-47f6-816f-bce368e018a5\" (UID: \"e6e40349-5135-47f6-816f-bce368e018a5\") "
Nov 02 20:14:02.660079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.660044    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca" pod "e6e40349-5135-47f6-816f-bce368e018a5" (UID: "e6e40349-5135-47f6-816f-bce368e018a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:02.660478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.660454    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:02.661814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.661766    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca" pod "e6e40349-5135-47f6-816f-bce368e018a5" (UID: "e6e40349-5135-47f6-816f-bce368e018a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:02.662454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.662431    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf" pod "e6e40349-5135-47f6-816f-bce368e018a5" (UID: "e6e40349-5135-47f6-816f-bce368e018a5"). InnerVolumeSpecName "kube-api-access-tmzwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:02.762504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.762407    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.762690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.762546    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765120    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765188    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765240    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765337    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765373    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765406    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765463    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzj4\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.765677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765644    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e40349-5135-47f6-816f-bce368e018a5-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:02.765733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.765671    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmzwf\" (UniqueName: \"kubernetes.io/projected/e6e40349-5135-47f6-816f-bce368e018a5-kube-api-access-tmzwf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:02.767303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.767275    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:14:02.866802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.866753    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.866991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.866842    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.866991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.866919    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.866991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.866958    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.866991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.866989    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867048    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzj4\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867150    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867236    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867321    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867329    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867551    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867637    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867818    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.867907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.867818    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.869551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.869527    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.870163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.870144    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.921664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.921636    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.921922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.921896    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzj4\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:02.949319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:02.949274    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:03.261346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.261304    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7569deda-479a-4eef-8175-b82491820747" path="/var/lib/kubelet/pods/7569deda-479a-4eef-8175-b82491820747/volumes"
Nov 02 20:14:03.368396 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:03.368337    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff507b91_f192_44ce_b32a_287ba00cc9c6.slice/crio-518b787f2073892adf05d7f7a89d11850cf6f91c5850d6c85720bbc7dacc7976 WatchSource:0}: Error finding container 518b787f2073892adf05d7f7a89d11850cf6f91c5850d6c85720bbc7dacc7976: Status 404 returned error can't find the container with id 518b787f2073892adf05d7f7a89d11850cf6f91c5850d6c85720bbc7dacc7976
Nov 02 20:14:03.369652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.369627    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:14:03.371487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.371453    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:03.371590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.371530    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:03.371590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.371574    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:03.438302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.438263    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" event={"ID":"ff507b91-f192-44ce-b32a-287ba00cc9c6","Type":"ContainerStarted","Data":"518b787f2073892adf05d7f7a89d11850cf6f91c5850d6c85720bbc7dacc7976"}
Nov 02 20:14:03.440224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.440193    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5" event={"ID":"e6e40349-5135-47f6-816f-bce368e018a5","Type":"ContainerDied","Data":"ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6"}
Nov 02 20:14:03.440350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.440248    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab340f079254f9244f3d48f10c1cd93168e10051bddea3e1fdcfd9bef245b0c6"
Nov 02 20:14:03.440350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:03.440258    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"
Nov 02 20:14:04.477380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:04.477317    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" event={"ID":"ff507b91-f192-44ce-b32a-287ba00cc9c6","Type":"ContainerStarted","Data":"68a14f04907ad7c9c8b22398b4057c5b0840cfd9bef4e4d2f1073a4ba42f3b2b"}
Nov 02 20:14:04.665805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:04.659859    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podStartSLOduration=2.659834955 podStartE2EDuration="2.659834955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:04.634870695 +0000 UTC m=+1333.509141585" watchObservedRunningTime="2026-11-02 20:14:04.659834955 +0000 UTC m=+1333.534105846"
Nov 02 20:14:04.953744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:04.950986    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:04.954443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:04.954079    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:04.954443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:04.954156    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:05.733260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.733227    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"]
Nov 02 20:14:05.751893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.745087    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.790676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.790639    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"]
Nov 02 20:14:05.811503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.811471    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzv4l\" (UniqueName: \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.811680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.811627    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.811736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.811683    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.873807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.834634    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:14:05.873807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.834642    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:14:05.913274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.913235    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.913457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.913359    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzv4l\" (UniqueName: \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.913507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.913469    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.914038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.914012    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.915827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.915807    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:05.950813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.950234    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:05.950813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.950296    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:05.971446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:05.971415    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzv4l\" (UniqueName: \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\") pod \"maas-api-revoke-sub-1422d13d-2cpx6\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:06.082517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:06.082416    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:06.517585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:06.515583    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"]
Nov 02 20:14:06.953803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:06.953059    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:06.953803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:06.953115    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:07.073376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.073311    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"]
Nov 02 20:14:07.078149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.078110    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2c1d43-2ssjv"]
Nov 02 20:14:07.290728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.290686    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f00c3c-682c-406d-a845-44dc4baaa920" path="/var/lib/kubelet/pods/f6f00c3c-682c-406d-a845-44dc4baaa920/volumes"
Nov 02 20:14:07.573904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.573806    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6" event={"ID":"f0be176b-40bd-43db-9f3a-2078822aa931","Type":"ContainerStarted","Data":"c1fde8b92950ab2dc04150d8d63140de10b67d9c0c12c78a38e682e8a7da4104"}
Nov 02 20:14:07.573904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.573867    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6" event={"ID":"f0be176b-40bd-43db-9f3a-2078822aa931","Type":"ContainerStarted","Data":"bcdbb994d9c6f8c47c4112e1a3f7dd820f5a21ec3b44dab7446e72a567b74dfd"}
Nov 02 20:14:07.686143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.686083    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6" podStartSLOduration=2.686068633 podStartE2EDuration="2.686068633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:07.682227571 +0000 UTC m=+1336.556498460" watchObservedRunningTime="2026-11-02 20:14:07.686068633 +0000 UTC m=+1336.560339552"
Nov 02 20:14:07.957807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.953850    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:07.957807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:07.953919    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:08.069931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:08.069885    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"]
Nov 02 20:14:08.074838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:08.074807    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d785604-9wmqv"]
Nov 02 20:14:08.632806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:08.622833    5099 generic.go:356] "Generic (PLEG): container finished" podID="f0be176b-40bd-43db-9f3a-2078822aa931" containerID="c1fde8b92950ab2dc04150d8d63140de10b67d9c0c12c78a38e682e8a7da4104" exitCode=0
Nov 02 20:14:08.632806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:08.622908    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6" event={"ID":"f0be176b-40bd-43db-9f3a-2078822aa931","Type":"ContainerDied","Data":"c1fde8b92950ab2dc04150d8d63140de10b67d9c0c12c78a38e682e8a7da4104"}
Nov 02 20:14:09.002465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:09.002186    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:09.003062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:09.003028    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:09.317986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:09.317731    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d3e622-6474-46a8-a673-5e0225c9a7d4" path="/var/lib/kubelet/pods/95d3e622-6474-46a8-a673-5e0225c9a7d4/volumes"
Nov 02 20:14:09.963051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:09.963016    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:09.963332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:09.963301    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:10.156923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.156877    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:10.244260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.233898    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:14:10.324881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.321900    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:10.440988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.440882    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:10.540198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.540135    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:10.540975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:10.540935    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:11.303846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.303771    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:11.556865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.556236    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:11.567807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.557161    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:11.567807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.557411    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:14:11.567807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.557775    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:14:11.567807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.558195    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:14:11.567807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.558269    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:11.665471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.635088    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:14:11.831857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.828821    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:11.831857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:11.828909    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:12.317809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.282687    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6" podUID="8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:12.329987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.329945    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:12.330250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.330217    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:12.372810 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:14:12.372756    5099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.042s"
Nov 02 20:14:12.632071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.632034    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:14:12.712208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.712173    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:14:12.762007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.761980    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:14:12.805805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.780295    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:14:12.967171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.967093    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:14:12.967420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.967399    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:14:12.993807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.806892    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:14:12.998930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.807404    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:14:13.014955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.807436    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:14:13.019036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.807452    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:14:13.019956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.821654    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:13.043356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.897034    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:13.043675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.966971    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:13.043893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.043860    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:13.044018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.807382    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:13.044674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:12.807354    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:14:13.045029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.026469    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:14:13.076255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.075474    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:14:13.170130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.170072    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\") pod \"f0be176b-40bd-43db-9f3a-2078822aa931\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") "
Nov 02 20:14:13.170304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.170223    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\" (UniqueName: \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\") pod \"f0be176b-40bd-43db-9f3a-2078822aa931\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") "
Nov 02 20:14:13.170493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.170470    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\") pod \"f0be176b-40bd-43db-9f3a-2078822aa931\" (UID: \"f0be176b-40bd-43db-9f3a-2078822aa931\") "
Nov 02 20:14:13.176174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.176146    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca" pod "f0be176b-40bd-43db-9f3a-2078822aa931" (UID: "f0be176b-40bd-43db-9f3a-2078822aa931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:13.179224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.179014    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l" pod "f0be176b-40bd-43db-9f3a-2078822aa931" (UID: "f0be176b-40bd-43db-9f3a-2078822aa931"). InnerVolumeSpecName "kube-api-access-tzv4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:13.179751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.179496    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca" pod "f0be176b-40bd-43db-9f3a-2078822aa931" (UID: "f0be176b-40bd-43db-9f3a-2078822aa931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:13.180861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.179925    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:14:13.277857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.277277    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzv4l\" (UniqueName: \"kubernetes.io/projected/f0be176b-40bd-43db-9f3a-2078822aa931-kube-api-access-tzv4l\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:13.277857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.277319    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:13.277857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.277336    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0be176b-40bd-43db-9f3a-2078822aa931-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:13.542412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.542320    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"
Nov 02 20:14:13.662890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.648271    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6" event={"ID":"f0be176b-40bd-43db-9f3a-2078822aa931","Type":"ContainerDied","Data":"bcdbb994d9c6f8c47c4112e1a3f7dd820f5a21ec3b44dab7446e72a567b74dfd"}
Nov 02 20:14:13.662890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.648371    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcdbb994d9c6f8c47c4112e1a3f7dd820f5a21ec3b44dab7446e72a567b74dfd"
Nov 02 20:14:13.683848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.681842    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:13.815766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.815671    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:13.960440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.960405    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 20:14:13.978839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:13.960722    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 20:14:14.001635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.001570    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:14:14.006062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.006026    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:14.069874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.038293    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:14.224812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.043851    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:14.224812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.044880    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:14.464239 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.464201    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:14:14.628966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.627223    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:14:14.822864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.822086    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:14:14.840174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.840121    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:14:14.859953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.851296    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:14:14.904760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.904707    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:14:14.906705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:14.905455    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:15.953090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:15.953048    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:14:15.953559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:15.953116    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:16.101135 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:14:16.101070    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:14:16.662136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:16.660630    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:16.662136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:16.661973    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:16.663875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:16.663823    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:14:18.905095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.905052    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"]
Nov 02 20:14:18.905938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.905916    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f0be176b-40bd-43db-9f3a-2078822aa931" containerName="revoke-keys"
Nov 02 20:14:18.905938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.905936    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0be176b-40bd-43db-9f3a-2078822aa931" containerName="revoke-keys"
Nov 02 20:14:18.906156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.906103    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0be176b-40bd-43db-9f3a-2078822aa931" containerName="revoke-keys"
Nov 02 20:14:18.911682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.911647    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:18.959342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:18.959301    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"]
Nov 02 20:14:19.000755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.000713    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:14:19.000755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.000722    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:14:19.005506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.005475    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5nd\" (UniqueName: \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.005618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.005525    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.005671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.005650    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.106701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.106663    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.106949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.106736    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.107072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.107056    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5nd\" (UniqueName: \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.107515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.107490    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.109199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.109180    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.161506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.161424    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5nd\" (UniqueName: \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\") pod \"maas-api-revoke-sub-1f16f285-59z8w\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.231923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.231890    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:19.619248 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:19.619200    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55875403_32d0_4ac0_a81c_08ea028493a2.slice/crio-918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d WatchSource:0}: Error finding container 918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d: Status 404 returned error can't find the container with id 918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d
Nov 02 20:14:19.620721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.620694    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"]
Nov 02 20:14:19.881687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:19.881595    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w" event={"ID":"55875403-32d0-4ac0-a81c-08ea028493a2","Type":"ContainerStarted","Data":"918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d"}
Nov 02 20:14:20.892008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:20.891974    5099 generic.go:356] "Generic (PLEG): container finished" podID="55875403-32d0-4ac0-a81c-08ea028493a2" containerID="2eac2bf66c956a9e081dcb187952821f90b7e2dfe7c8f4ed89ca52fc362b35ab" exitCode=0
Nov 02 20:14:20.892381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:20.892052    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w" event={"ID":"55875403-32d0-4ac0-a81c-08ea028493a2","Type":"ContainerDied","Data":"2eac2bf66c956a9e081dcb187952821f90b7e2dfe7c8f4ed89ca52fc362b35ab"}
Nov 02 20:14:22.073945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.073917    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:22.176542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.176442    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\") pod \"55875403-32d0-4ac0-a81c-08ea028493a2\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") "
Nov 02 20:14:22.176732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.176659    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\" (UniqueName: \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\") pod \"55875403-32d0-4ac0-a81c-08ea028493a2\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") "
Nov 02 20:14:22.176732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.176724    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\") pod \"55875403-32d0-4ac0-a81c-08ea028493a2\" (UID: \"55875403-32d0-4ac0-a81c-08ea028493a2\") "
Nov 02 20:14:22.176876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.176778    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca" pod "55875403-32d0-4ac0-a81c-08ea028493a2" (UID: "55875403-32d0-4ac0-a81c-08ea028493a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:22.177436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.177402    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:22.179340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.179278    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd" pod "55875403-32d0-4ac0-a81c-08ea028493a2" (UID: "55875403-32d0-4ac0-a81c-08ea028493a2"). InnerVolumeSpecName "kube-api-access-vf5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:22.179340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.179300    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca" pod "55875403-32d0-4ac0-a81c-08ea028493a2" (UID: "55875403-32d0-4ac0-a81c-08ea028493a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:22.278463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.278427    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5nd\" (UniqueName: \"kubernetes.io/projected/55875403-32d0-4ac0-a81c-08ea028493a2-kube-api-access-vf5nd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:22.278463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.278460    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55875403-32d0-4ac0-a81c-08ea028493a2-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:22.911142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.911114    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"
Nov 02 20:14:22.911354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.911172    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w" event={"ID":"55875403-32d0-4ac0-a81c-08ea028493a2","Type":"ContainerDied","Data":"918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d"}
Nov 02 20:14:22.911354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:22.911231    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="918d7ca55bff081f303bbd240c2532460baf8a28da19640960d9dd2e9ada683d"
Nov 02 20:14:23.046332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.046295    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"]
Nov 02 20:14:23.051379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.051350    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6680050c-7wl6w"]
Nov 02 20:14:23.262036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.261941    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68793c29-8023-467c-9541-4a7eb73792df" path="/var/lib/kubelet/pods/68793c29-8023-467c-9541-4a7eb73792df/volumes"
Nov 02 20:14:23.351114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.351082    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"]
Nov 02 20:14:23.351778 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.351763    5099 cpu_manager.go:414] "containerMap: removing container" podUID="55875403-32d0-4ac0-a81c-08ea028493a2" containerName="revoke-keys"
Nov 02 20:14:23.351845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.351793    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55875403-32d0-4ac0-a81c-08ea028493a2" containerName="revoke-keys"
Nov 02 20:14:23.351936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.351926    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="55875403-32d0-4ac0-a81c-08ea028493a2" containerName="revoke-keys"
Nov 02 20:14:23.356003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.355984    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.405554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.405524    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:14:23.405714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.405540    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:14:23.407257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.407234    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"]
Nov 02 20:14:23.494443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.494411    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.494614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.494467    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6qc\" (UniqueName: \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.494614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.494594    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.596378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.596297    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.596378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.596349    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6qc\" (UniqueName: \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.596575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.596389    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.597129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.597101    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.598690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.598668    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.650976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.650942    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6qc\" (UniqueName: \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\") pod \"maas-api-revoke-sub-9dad84ad-dkjcl\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:23.671043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:23.671010    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:24.051034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:24.051002    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"]
Nov 02 20:14:24.052481 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:24.052454    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92764572_a2b5_4a56_a0f2_b9195609440c.slice/crio-d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6 WatchSource:0}: Error finding container d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6: Status 404 returned error can't find the container with id d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6
Nov 02 20:14:24.928509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:24.928471    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl" event={"ID":"92764572-a2b5-4a56-a0f2-b9195609440c","Type":"ContainerStarted","Data":"317c244de885aa8dadb7039e575cedbff4a149689731c4b016730e8c7cfe832d"}
Nov 02 20:14:24.928935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:24.928531    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl" event={"ID":"92764572-a2b5-4a56-a0f2-b9195609440c","Type":"ContainerStarted","Data":"d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6"}
Nov 02 20:14:25.035373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.035312    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl" podStartSLOduration=2.035295191 podStartE2EDuration="2.035295191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:25.034887713 +0000 UTC m=+1353.909158601" watchObservedRunningTime="2026-11-02 20:14:25.035295191 +0000 UTC m=+1353.909566080"
Nov 02 20:14:25.093244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.092826    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"]
Nov 02 20:14:25.099697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.099652    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67b6247-r7vz5"]
Nov 02 20:14:25.261946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.261853    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b171bf4-2035-43d6-833e-1efab58caf0d" path="/var/lib/kubelet/pods/7b171bf4-2035-43d6-833e-1efab58caf0d/volumes"
Nov 02 20:14:25.940076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.940037    5099 generic.go:356] "Generic (PLEG): container finished" podID="92764572-a2b5-4a56-a0f2-b9195609440c" containerID="317c244de885aa8dadb7039e575cedbff4a149689731c4b016730e8c7cfe832d" exitCode=0
Nov 02 20:14:25.940486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:25.940123    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl" event={"ID":"92764572-a2b5-4a56-a0f2-b9195609440c","Type":"ContainerDied","Data":"317c244de885aa8dadb7039e575cedbff4a149689731c4b016730e8c7cfe832d"}
Nov 02 20:14:27.048835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.048799    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"]
Nov 02 20:14:27.092507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.092471    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b97716-7f4tj"]
Nov 02 20:14:27.149483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.149457    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:27.253739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.253654    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\") pod \"92764572-a2b5-4a56-a0f2-b9195609440c\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") "
Nov 02 20:14:27.253739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.253714    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\" (UniqueName: \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\") pod \"92764572-a2b5-4a56-a0f2-b9195609440c\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") "
Nov 02 20:14:27.254032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.253889    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\") pod \"92764572-a2b5-4a56-a0f2-b9195609440c\" (UID: \"92764572-a2b5-4a56-a0f2-b9195609440c\") "
Nov 02 20:14:27.254096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.254059    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca" pod "92764572-a2b5-4a56-a0f2-b9195609440c" (UID: "92764572-a2b5-4a56-a0f2-b9195609440c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:27.254512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.254463    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:27.256360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.256334    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca" pod "92764572-a2b5-4a56-a0f2-b9195609440c" (UID: "92764572-a2b5-4a56-a0f2-b9195609440c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:27.256469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.256427    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc" pod "92764572-a2b5-4a56-a0f2-b9195609440c" (UID: "92764572-a2b5-4a56-a0f2-b9195609440c"). InnerVolumeSpecName "kube-api-access-rc6qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:27.265985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.265952    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ca2e4c-d9a4-4c10-b64f-8eea013762b6" path="/var/lib/kubelet/pods/32ca2e4c-d9a4-4c10-b64f-8eea013762b6/volumes"
Nov 02 20:14:27.356620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.356581    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6qc\" (UniqueName: \"kubernetes.io/projected/92764572-a2b5-4a56-a0f2-b9195609440c-kube-api-access-rc6qc\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:27.356620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.356620    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92764572-a2b5-4a56-a0f2-b9195609440c-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:27.963767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.963729    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl" event={"ID":"92764572-a2b5-4a56-a0f2-b9195609440c","Type":"ContainerDied","Data":"d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6"}
Nov 02 20:14:27.963767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.963753    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"
Nov 02 20:14:27.964023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:27.963809    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37eadcd9390fa45dd89626f5a1a6af03a1c1d92f6cbfd4ec9ee5757bf8f89e6"
Nov 02 20:14:30.565944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.565907    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"]
Nov 02 20:14:30.566687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.566665    5099 cpu_manager.go:414] "containerMap: removing container" podUID="92764572-a2b5-4a56-a0f2-b9195609440c" containerName="revoke-keys"
Nov 02 20:14:30.566687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.566684    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92764572-a2b5-4a56-a0f2-b9195609440c" containerName="revoke-keys"
Nov 02 20:14:30.566926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.566897    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="92764572-a2b5-4a56-a0f2-b9195609440c" containerName="revoke-keys"
Nov 02 20:14:30.571294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.571274    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.621728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.621688    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"]
Nov 02 20:14:30.662334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.662299    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:14:30.663066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.663042    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:14:30.708731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.708690    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.708934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.708757    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.708993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.708945    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896b7\" (UniqueName: \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.810546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.810503    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896b7\" (UniqueName: \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.810809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.810759    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.810868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.810853    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.811463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.811436    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.813310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.813281    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.865335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.865304    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896b7\" (UniqueName: \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\") pod \"maas-api-revoke-sub-f397d02e-pp826\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:30.889195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:30.889162    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:31.265429 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:31.265399    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f4a7ea_2577_4983_9ae4_0f4209c34e9d.slice/crio-c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04 WatchSource:0}: Error finding container c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04: Status 404 returned error can't find the container with id c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04
Nov 02 20:14:31.266777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:31.266754    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"]
Nov 02 20:14:31.897687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:31.897587    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:14:31.902339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:31.902314    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:31.995679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:31.995621    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:14:32.008309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.008258    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826" event={"ID":"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d","Type":"ContainerStarted","Data":"e357f403f410ad0bd4d3ba77254dcc10055976faaae0265a4d6ac5ff851f3fad"}
Nov 02 20:14:32.008499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.008323    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826" event={"ID":"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d","Type":"ContainerStarted","Data":"c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04"}
Nov 02 20:14:32.028537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.028496    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7j6\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.029950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.029916    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.029990    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030119    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030163    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030222    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030308    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.030454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.030350    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.131692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131646    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.131920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131713    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.131920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131828    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.131920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131849    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.131920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131905    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131938    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.131994    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132035    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132143    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7j6\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132431    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132586    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132640    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132720    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.132884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.132867    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.134372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.134355    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.134603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.134581    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.144875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.144813    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826" podStartSLOduration=2.144774656 podStartE2EDuration="2.144774656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:32.142278737 +0000 UTC m=+1361.016549627" watchObservedRunningTime="2026-11-02 20:14:32.144774656 +0000 UTC m=+1361.019045548"
Nov 02 20:14:32.185824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.185743    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.186200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.186179    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7j6\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:32.218645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:32.218539    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:33.203817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.196255    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:14:33.314686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.314618    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:33.315044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.315015    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:33.319934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.319903    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:33.539368 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.539311    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:33.670284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.670243    5099 generic.go:356] "Generic (PLEG): container finished" podID="f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" containerID="e357f403f410ad0bd4d3ba77254dcc10055976faaae0265a4d6ac5ff851f3fad" exitCode=0
Nov 02 20:14:33.713260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.713156    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" event={"ID":"6068698f-2e9f-4167-94be-ba4a3e8a78ee","Type":"ContainerStarted","Data":"74ff60855871c128addb18cf90904e34a56abaac3c00eecd7723e2902df56cff"}
Nov 02 20:14:33.713991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.713967    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:14:33.737195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.722213    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:14:33.737195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.722272    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826" event={"ID":"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d","Type":"ContainerDied","Data":"e357f403f410ad0bd4d3ba77254dcc10055976faaae0265a4d6ac5ff851f3fad"}
Nov 02 20:14:33.737195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.722388    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.870755    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898278    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898340    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898367    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898480    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898530    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.898808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898767    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.899385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898825    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nbk\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:33.899385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:33.898911    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.002798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.002423    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.029093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.029063    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.032311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.032283    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.032978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.032957    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.055807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.033302    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.057674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.057647    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nbk\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.058349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.008254    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.059386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.030225    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.059549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.056060    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:34.060056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.056882    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:34.060056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.056911    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Nov 02 20:14:34.060056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.035125    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.060056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.058542    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.060341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.060174    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.060341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.060220    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.064857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.059302    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.064857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.060931    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.089804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.073707    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:34.089804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.073866    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:34.089804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.088038    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:34.089804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.089681    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:14:34.092214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.091726    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.116894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.116195    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:34.330153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.314866    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.432747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.431631    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.481808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.461918    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nbk\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.709118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.709075    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:34.798929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.798889    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"]
Nov 02 20:14:34.925883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:34.925851    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:35.021808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:35.018826    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"]
Nov 02 20:14:35.308151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:35.308051    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:35.317356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:35.310897    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:35.485765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:35.484531    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:35.985306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:35.985229    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:36.012810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.008988    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:36.024481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.024431    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:36.139805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.065635    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cpf\" (UniqueName: \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.163806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.140382    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.164184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.128309    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:36.217533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.204314    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.312545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.312492    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:36.515890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.491001    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:36.516717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.516675    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:14:36.663813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.640674    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.663813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.640817    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.663813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.640944    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cpf\" (UniqueName: \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:36.663813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:36.643170    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:37.040808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.035303    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:37.049808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.046853    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:37.049808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.047237    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:37.081807 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:14:37.072385    5099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.784s"
Nov 02 20:14:37.081807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.072450    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" event={"ID":"6068698f-2e9f-4167-94be-ba4a3e8a78ee","Type":"ContainerStarted","Data":"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"}
Nov 02 20:14:37.081807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.072632    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:14:37.152807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.148971    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:14:37.160905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.160855    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"]
Nov 02 20:14:37.185942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.185907    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:14:37.186198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.186181    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:14:37.186364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.186350    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:14:37.186502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.186490    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:14:37.186632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.186621    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:14:37.189805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.186730    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:14:37.234691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.234599    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:37.381172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.381136    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:37.381486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.381438    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:37.462945 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:37.444855    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca717e9b_8d99_4caa_bf89_fd40b1ae6a64.slice/crio-75719898116bab9c452578fc126e4da7a8d4a6d1058a4a5e2838423032862e61 WatchSource:0}: Error finding container 75719898116bab9c452578fc126e4da7a8d4a6d1058a4a5e2838423032862e61: Status 404 returned error can't find the container with id 75719898116bab9c452578fc126e4da7a8d4a6d1058a4a5e2838423032862e61
Nov 02 20:14:37.532028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.523064    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cpf\" (UniqueName: \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\") pod \"maas-api-revoke-sub-45fb40bd-n2ggr\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:37.532028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.523200    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:37.532028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.523457    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:14:37.625843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.611310    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:37.714718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.714682    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:37.715294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.715259    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:37.767571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.767142    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:37.928820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.893612    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:37.928820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.893739    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:37.928820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.893804    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:37.934226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:37.931300    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:14:38.092994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.092891    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:38.096282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.096166    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:14:38.192632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.147082    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:14:38.229125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.148913    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:38.230628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.230603    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:14:38.341432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.291175    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:38.341986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.341950    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:38.628951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.628914    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:14:38.733842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.731009    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" event={"ID":"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64","Type":"ContainerStarted","Data":"75719898116bab9c452578fc126e4da7a8d4a6d1058a4a5e2838423032862e61"}
Nov 02 20:14:38.810025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.809986    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:38.830256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.817846    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6665dcd4-vthj2"]
Nov 02 20:14:38.896859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.878974    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:14:38.968919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.968844    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"]
Nov 02 20:14:38.984540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.984515    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec65fcaa-xpp2g"]
Nov 02 20:14:38.997933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.987099    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podStartSLOduration=7.987079512 podStartE2EDuration="7.987079512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:37.539180821 +0000 UTC m=+1366.413451711" watchObservedRunningTime="2026-11-02 20:14:38.987079512 +0000 UTC m=+1367.861350402"
Nov 02 20:14:39.000756 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:38.998123    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:14:39.018830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.016768    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:14:39.097170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.032203    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:14:39.097170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.036136    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:14:39.109936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.109898    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"]
Nov 02 20:14:39.115649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.115622    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.122217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.122192    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"]
Nov 02 20:14:39.220548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.220439    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:39.220548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.220493    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:39.299761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.264907    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.299761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.265106    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nrk\" (UniqueName: \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.299761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.265255    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.321006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.320973    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:39.323918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.323890    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02406e10-023c-4d34-aaec-ec3fd2ddf34c" path="/var/lib/kubelet/pods/02406e10-023c-4d34-aaec-ec3fd2ddf34c/volumes"
Nov 02 20:14:39.324524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.324501    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563f5405-288d-49e2-a436-d000fddd2e81" path="/var/lib/kubelet/pods/563f5405-288d-49e2-a436-d000fddd2e81/volumes"
Nov 02 20:14:39.387168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.387122    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nrk\" (UniqueName: \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.387648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.387623    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.387915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.387894    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.389145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.389107    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.391184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.391136    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.493871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.490577    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\") pod \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") "
Nov 02 20:14:39.493871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.490994    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\") pod \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") "
Nov 02 20:14:39.493871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.491420    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\" (UniqueName: \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\") pod \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\" (UID: \"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d\") "
Nov 02 20:14:39.493871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.491684    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca" pod "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" (UID: "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:39.493871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.494916    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:39.497930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.497906    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"]
Nov 02 20:14:39.499066 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:39.499036    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2152e865_e5c9_4e2a_995f_ce1eb7044d2a.slice/crio-88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2 WatchSource:0}: Error finding container 88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2: Status 404 returned error can't find the container with id 88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2
Nov 02 20:14:39.499259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.499227    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca" pod "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" (UID: "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:39.499728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.499704    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7" pod "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" (UID: "f3f4a7ea-2577-4983-9ae4-0f4209c34e9d"). InnerVolumeSpecName "kube-api-access-896b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:39.503179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.503157    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nrk\" (UniqueName: \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\") pod \"maas-api-revoke-sub-42466ff3-phc89\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.509596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.509569    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:39.596675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.596643    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896b7\" (UniqueName: \"kubernetes.io/projected/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-kube-api-access-896b7\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:39.596835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.596680    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:39.778957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.778859    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr" event={"ID":"2152e865-e5c9-4e2a-995f-ce1eb7044d2a","Type":"ContainerStarted","Data":"88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2"}
Nov 02 20:14:39.783606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.783509    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826" event={"ID":"f3f4a7ea-2577-4983-9ae4-0f4209c34e9d","Type":"ContainerDied","Data":"c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04"}
Nov 02 20:14:39.783606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.783545    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"
Nov 02 20:14:39.783606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.783566    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c862bf0149d93e27966d39b76c5d2005f5524a2174df9683ff8f31b8d0647e04"
Nov 02 20:14:39.788258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:39.788230    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" event={"ID":"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64","Type":"ContainerStarted","Data":"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"}
Nov 02 20:14:40.011468 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:40.011435    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e563929_5245_4353_a999_6271c9309709.slice/crio-5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702 WatchSource:0}: Error finding container 5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702: Status 404 returned error can't find the container with id 5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702
Nov 02 20:14:40.013585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.013555    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"]
Nov 02 20:14:40.279809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.262900    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:40.279809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.262963    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:40.393420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.393331    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podStartSLOduration=7.393306366 podStartE2EDuration="7.393306366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:40.378516747 +0000 UTC m=+1369.252787636" watchObservedRunningTime="2026-11-02 20:14:40.393306366 +0000 UTC m=+1369.267577256"
Nov 02 20:14:40.719211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.719185    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:40.720587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.720564    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:40.720658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.720605    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:40.808862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.808824    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89" event={"ID":"5e563929-5245-4353-a999-6271c9309709","Type":"ContainerStarted","Data":"424d3692c8996e960800e3fb27c023eed834495cdcee3402d9574476db418759"}
Nov 02 20:14:40.809363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.809322    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89" event={"ID":"5e563929-5245-4353-a999-6271c9309709","Type":"ContainerStarted","Data":"5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702"}
Nov 02 20:14:40.810959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.810932    5099 generic.go:356] "Generic (PLEG): container finished" podID="2152e865-e5c9-4e2a-995f-ce1eb7044d2a" containerID="61d15c6e8836eb4664d988a204aa1f92b2587933573445887506360523ea4aa8" exitCode=0
Nov 02 20:14:40.811511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.811491    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr" event={"ID":"2152e865-e5c9-4e2a-995f-ce1eb7044d2a","Type":"ContainerDied","Data":"61d15c6e8836eb4664d988a204aa1f92b2587933573445887506360523ea4aa8"}
Nov 02 20:14:40.922324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:40.922195    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89" podStartSLOduration=3.922180576 podStartE2EDuration="3.922180576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:40.917935071 +0000 UTC m=+1369.792205960" watchObservedRunningTime="2026-11-02 20:14:40.922180576 +0000 UTC m=+1369.796451464"
Nov 02 20:14:41.219641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.219609    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:41.219877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.219663    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:41.719631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.719590    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:41.719850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.719646    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:41.822654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.822622    5099 generic.go:356] "Generic (PLEG): container finished" podID="5e563929-5245-4353-a999-6271c9309709" containerID="424d3692c8996e960800e3fb27c023eed834495cdcee3402d9574476db418759" exitCode=0
Nov 02 20:14:41.823130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:41.822708    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89" event={"ID":"5e563929-5245-4353-a999-6271c9309709","Type":"ContainerDied","Data":"424d3692c8996e960800e3fb27c023eed834495cdcee3402d9574476db418759"}
Nov 02 20:14:42.222598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.222561    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:42.223680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.223128    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:42.506498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.506425    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:42.619751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.619708    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\") pod \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") "
Nov 02 20:14:42.619973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.619771    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\" (UniqueName: \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\") pod \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") "
Nov 02 20:14:42.619973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.619909    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\") pod \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\" (UID: \"2152e865-e5c9-4e2a-995f-ce1eb7044d2a\") "
Nov 02 20:14:42.621020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.620979    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca" pod "2152e865-e5c9-4e2a-995f-ce1eb7044d2a" (UID: "2152e865-e5c9-4e2a-995f-ce1eb7044d2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:42.624703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.624326    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca" pod "2152e865-e5c9-4e2a-995f-ce1eb7044d2a" (UID: "2152e865-e5c9-4e2a-995f-ce1eb7044d2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:42.624703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.624540    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf" pod "2152e865-e5c9-4e2a-995f-ce1eb7044d2a" (UID: "2152e865-e5c9-4e2a-995f-ce1eb7044d2a"). InnerVolumeSpecName "kube-api-access-x9cpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:42.720367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.720321    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:42.720525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.720378    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:42.721913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.721883    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.721913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.721915    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.722089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.721929    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cpf\" (UniqueName: \"kubernetes.io/projected/2152e865-e5c9-4e2a-995f-ce1eb7044d2a-kube-api-access-x9cpf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:42.804018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.803870    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"]
Nov 02 20:14:42.806343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.806313    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2152e865-e5c9-4e2a-995f-ce1eb7044d2a" containerName="revoke-keys"
Nov 02 20:14:42.806509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.806495    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2152e865-e5c9-4e2a-995f-ce1eb7044d2a" containerName="revoke-keys"
Nov 02 20:14:42.806653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.806641    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" containerName="revoke-keys"
Nov 02 20:14:42.806725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.806715    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" containerName="revoke-keys"
Nov 02 20:14:42.807051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.807035    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2152e865-e5c9-4e2a-995f-ce1eb7044d2a" containerName="revoke-keys"
Nov 02 20:14:42.807154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.807145    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" containerName="revoke-keys"
Nov 02 20:14:42.812266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.812245    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"]
Nov 02 20:14:42.812489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.812472    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:42.904812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.903124    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"
Nov 02 20:14:42.904812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.903882    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr" event={"ID":"2152e865-e5c9-4e2a-995f-ce1eb7044d2a","Type":"ContainerDied","Data":"88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2"}
Nov 02 20:14:42.904812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.903944    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88be1db975d6a55ef099172be437be06b8ee2cbcfcd591a888610ad3595b0ce2"
Nov 02 20:14:42.924728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.924695    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:42.924960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.924831    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cgz\" (UniqueName: \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:42.925043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:42.924994    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.026773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.026732    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.026975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.026928    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.027038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.027001    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cgz\" (UniqueName: \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.028010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.027978    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.029735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.029278    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.118414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.111307    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cgz\" (UniqueName: \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\") pod \"maas-api-revoke-sub-07182027-9n5rm\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.131880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.130866    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:43.219312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.219275    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:43.219729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.219701    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:43.315315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.315289    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:43.416958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.415032    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\") pod \"5e563929-5245-4353-a999-6271c9309709\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") "
Nov 02 20:14:43.416958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.415205    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\" (UniqueName: \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\") pod \"5e563929-5245-4353-a999-6271c9309709\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") "
Nov 02 20:14:43.416958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.415306    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\") pod \"5e563929-5245-4353-a999-6271c9309709\" (UID: \"5e563929-5245-4353-a999-6271c9309709\") "
Nov 02 20:14:43.417483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.417433    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca" pod "5e563929-5245-4353-a999-6271c9309709" (UID: "5e563929-5245-4353-a999-6271c9309709"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:43.420243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.420215    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk" pod "5e563929-5245-4353-a999-6271c9309709" (UID: "5e563929-5245-4353-a999-6271c9309709"). InnerVolumeSpecName "kube-api-access-t9nrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:43.515107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.515028    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca" pod "5e563929-5245-4353-a999-6271c9309709" (UID: "5e563929-5245-4353-a999-6271c9309709"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:43.517646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.517616    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e563929-5245-4353-a999-6271c9309709-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:43.517880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.517650    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9nrk\" (UniqueName: \"kubernetes.io/projected/5e563929-5245-4353-a999-6271c9309709-kube-api-access-t9nrk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:43.517880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.517666    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e563929-5245-4353-a999-6271c9309709-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:43.719729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.719639    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:43.719729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.719702    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:43.727662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.727629    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"]
Nov 02 20:14:43.728161 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:43.728136    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f5e60a_99fc_45ff_97ce_1949bacd9159.slice/crio-832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818 WatchSource:0}: Error finding container 832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818: Status 404 returned error can't find the container with id 832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818
Nov 02 20:14:43.924283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.924248    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"
Nov 02 20:14:43.926320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.924275    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89" event={"ID":"5e563929-5245-4353-a999-6271c9309709","Type":"ContainerDied","Data":"5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702"}
Nov 02 20:14:43.926320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.926391    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d96a1ae5a41301e232a530c3d0b42ed2529ad888b19c7ae084a5f1720cae702"
Nov 02 20:14:43.932406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:43.932358    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm" event={"ID":"83f5e60a-99fc-45ff-97ce-1949bacd9159","Type":"ContainerStarted","Data":"832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818"}
Nov 02 20:14:44.219953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:44.219917    5099 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 20:14:44.220129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:44.219981    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 20:14:44.720572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:44.720074    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:44.720572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:44.720133    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:45.005470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.005382    5099 generic.go:356] "Generic (PLEG): container finished" podID="83f5e60a-99fc-45ff-97ce-1949bacd9159" containerID="347916da24bd1496bca3b9e5f6123cd1e616e78f0194709cb06eed203bf544fc" exitCode=0
Nov 02 20:14:45.006159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.005844    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm" event={"ID":"83f5e60a-99fc-45ff-97ce-1949bacd9159","Type":"ContainerDied","Data":"347916da24bd1496bca3b9e5f6123cd1e616e78f0194709cb06eed203bf544fc"}
Nov 02 20:14:45.313370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.313206    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:45.314076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.314051    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:45.317417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.317388    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:14:45.720182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.720146    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 20:14:45.720345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:45.720206    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 20:14:46.110700 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:14:46.110659    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:14:46.217728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.217701    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:46.319794 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.319694    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\" (UniqueName: \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\") pod \"83f5e60a-99fc-45ff-97ce-1949bacd9159\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") "
Nov 02 20:14:46.319794 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.319741    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\") pod \"83f5e60a-99fc-45ff-97ce-1949bacd9159\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") "
Nov 02 20:14:46.320035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.319884    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\") pod \"83f5e60a-99fc-45ff-97ce-1949bacd9159\" (UID: \"83f5e60a-99fc-45ff-97ce-1949bacd9159\") "
Nov 02 20:14:46.320195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.320168    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca" pod "83f5e60a-99fc-45ff-97ce-1949bacd9159" (UID: "83f5e60a-99fc-45ff-97ce-1949bacd9159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:46.320503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.320469    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:46.322090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.322065    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz" pod "83f5e60a-99fc-45ff-97ce-1949bacd9159" (UID: "83f5e60a-99fc-45ff-97ce-1949bacd9159"). InnerVolumeSpecName "kube-api-access-j4cgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:46.322193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.322096    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca" pod "83f5e60a-99fc-45ff-97ce-1949bacd9159" (UID: "83f5e60a-99fc-45ff-97ce-1949bacd9159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:46.421636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.421595    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cgz\" (UniqueName: \"kubernetes.io/projected/83f5e60a-99fc-45ff-97ce-1949bacd9159-kube-api-access-j4cgz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:46.421636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:46.421633    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f5e60a-99fc-45ff-97ce-1949bacd9159-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:47.026723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.026694    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"
Nov 02 20:14:47.026958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.026878    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm" event={"ID":"83f5e60a-99fc-45ff-97ce-1949bacd9159","Type":"ContainerDied","Data":"832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818"}
Nov 02 20:14:47.026958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.026923    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832f44053f0b33c1e06846d84b0787a4249365a6dc3df667e4db2e5814740818"
Nov 02 20:14:47.185170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.185132    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:47.185992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.185928    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:47.189434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:47.189388    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:14:48.932630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.932598    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:14:48.933377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933357    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5e563929-5245-4353-a999-6271c9309709" containerName="revoke-keys"
Nov 02 20:14:48.933432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933380    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e563929-5245-4353-a999-6271c9309709" containerName="revoke-keys"
Nov 02 20:14:48.933432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933430    5099 cpu_manager.go:414] "containerMap: removing container" podUID="83f5e60a-99fc-45ff-97ce-1949bacd9159" containerName="revoke-keys"
Nov 02 20:14:48.933524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933437    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f5e60a-99fc-45ff-97ce-1949bacd9159" containerName="revoke-keys"
Nov 02 20:14:48.933667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933652    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f5e60a-99fc-45ff-97ce-1949bacd9159" containerName="revoke-keys"
Nov 02 20:14:48.933701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.933683    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e563929-5245-4353-a999-6271c9309709" containerName="revoke-keys"
Nov 02 20:14:48.937747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:48.937723    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.025813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.025758    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:14:49.066527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066485    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.066527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066529    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.066814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066642    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5tnq\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.066814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066736    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.066967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066882    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.066967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066914    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.067069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.066973    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.067069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.067023    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.067168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.067134    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168316    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168408    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168436    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168469    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5tnq\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168550    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168658    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168688    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168768    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.168962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168901    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.169015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.168981    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.169069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.169026    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.169122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.169083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.169435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.169411    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.169560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.169543    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.170843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.170824    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.171435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.171417    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.222864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.222768    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.223006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.222867    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5tnq\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.255582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.255554    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:49.678596 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:14:49.678555    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f327d56_dbdf_49fd_a8ec_d29189565296.slice/crio-0f140b2a33c6104d82a9670b1c32cb2043aa3262bc5eeae8a78245f4ebf646c0 WatchSource:0}: Error finding container 0f140b2a33c6104d82a9670b1c32cb2043aa3262bc5eeae8a78245f4ebf646c0: Status 404 returned error can't find the container with id 0f140b2a33c6104d82a9670b1c32cb2043aa3262bc5eeae8a78245f4ebf646c0
Nov 02 20:14:49.680164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.680136    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:14:49.681461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.681425    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:49.681557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.681496    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:49.681557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:49.681530    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:14:50.056530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.056178    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" event={"ID":"9f327d56-dbdf-49fd-a8ec-d29189565296","Type":"ContainerStarted","Data":"5d38b682a7008c1c841f8314a0f3038bcb08e6a7264d1027916715784adb25fa"}
Nov 02 20:14:50.056530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.056244    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" event={"ID":"9f327d56-dbdf-49fd-a8ec-d29189565296","Type":"ContainerStarted","Data":"0f140b2a33c6104d82a9670b1c32cb2043aa3262bc5eeae8a78245f4ebf646c0"}
Nov 02 20:14:50.166382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.166311    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podStartSLOduration=2.1662969739999998 podStartE2EDuration="2.166296974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:50.16602387 +0000 UTC m=+1379.040294757" watchObservedRunningTime="2026-11-02 20:14:50.166296974 +0000 UTC m=+1379.040567859"
Nov 02 20:14:50.256613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.256537    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:50.260755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.260392    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:50.260755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:50.260457    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:51.085098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.085061    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"]
Nov 02 20:14:51.103433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.103401    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b328f656-8qfp7"]
Nov 02 20:14:51.265604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.265567    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:51.265987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.265954    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:51.312806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.298638    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeb9950-109f-406f-990f-5ea2387c5e43" path="/var/lib/kubelet/pods/dbeb9950-109f-406f-990f-5ea2387c5e43/volumes"
Nov 02 20:14:51.571564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.571536    5099 scope.go:122] "RemoveContainer" containerID="e4a95cba23899db03e9624a7c630d59560f78ed008f29492741fbcc649147e67"
Nov 02 20:14:51.582805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.582773    5099 scope.go:122] "RemoveContainer" containerID="9b31b67e414c059792710005c5f35e17a459ee12311c7f10c0c725f5e965be06"
Nov 02 20:14:51.593429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.593405    5099 scope.go:122] "RemoveContainer" containerID="85ee8f2df415f52c68fd9a334bdf85a650ff8334171c01838b03c56e36f17b90"
Nov 02 20:14:51.602933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.602913    5099 scope.go:122] "RemoveContainer" containerID="3d9601125310eae0b865b131a212dbf046244d2f46eb19b6afc0a61308a2849c"
Nov 02 20:14:51.612748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.612727    5099 scope.go:122] "RemoveContainer" containerID="c42ce5b91b56d504b9c218c91f1996b4c11b201dd390ed06259d63af24f9679d"
Nov 02 20:14:51.623198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.623177    5099 scope.go:122] "RemoveContainer" containerID="42ccbbac221925c1379a98035f8fa951d539a0a8fba0ae4f4e4083189946cdb0"
Nov 02 20:14:51.633254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.633234    5099 scope.go:122] "RemoveContainer" containerID="a31ad549decaf8b467881ef9a0c9b5873006c404923692a669dac8c86444104b"
Nov 02 20:14:51.668221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.668195    5099 scope.go:122] "RemoveContainer" containerID="aaca9ad48d485f990d1a3647ed25e922e12d4a58d10e2be736584405763dd7f6"
Nov 02 20:14:51.680703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:51.680680    5099 scope.go:122] "RemoveContainer" containerID="b71338c07104f1d6279e7432f763a4c25814d43a6c9e0bd0ce32ffac03e4ad3c"
Nov 02 20:14:52.066736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.066575    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"]
Nov 02 20:14:52.093422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.093388    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68d95df-dfj2w"]
Nov 02 20:14:52.257674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.257638    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:52.265385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.264928    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:52.269378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.269345    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"]
Nov 02 20:14:52.276241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.276220    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.303867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.303833    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"]
Nov 02 20:14:52.325642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.325553    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:14:52.325642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.325609    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:14:52.331551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.331528    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfbn\" (UniqueName: \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.331704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.331689    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.331746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.331725    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.432743 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.432702    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.432950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.432767    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.433006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.432969    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfbn\" (UniqueName: \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.433466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.433439    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.435078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.435053    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.487421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.487377    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfbn\" (UniqueName: \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\") pod \"maas-api-revoke-sub-5eb04643-bh2dc\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:52.590742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:52.590652    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:53.006953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:53.005942    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"]
Nov 02 20:14:53.120537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:53.120499    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc" event={"ID":"fa6f89fd-4c0f-461a-9060-129801d6bafd","Type":"ContainerStarted","Data":"b6d52820a1efecab3904e182672bd69310799fe62e0ee4529809ab5da94a51e2"}
Nov 02 20:14:53.259652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:53.259223    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:53.259652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:53.259300    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:53.278085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:53.278053    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7b7068-d11b-4670-be25-96a1d1916267" path="/var/lib/kubelet/pods/0b7b7068-d11b-4670-be25-96a1d1916267/volumes"
Nov 02 20:14:54.015114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.015082    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"]
Nov 02 20:14:54.020358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.020337    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.084993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.084951    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"]
Nov 02 20:14:54.087229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.087190    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.087630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.087612    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkqm\" (UniqueName: \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.087849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.087833    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.143812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.136403    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc" event={"ID":"fa6f89fd-4c0f-461a-9060-129801d6bafd","Type":"ContainerStarted","Data":"9ff485d2cc23871ba06a19d0a451160372c742ba31e090fa00bee9a978981536"}
Nov 02 20:14:54.192852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.192807    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.193075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.193053    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkqm\" (UniqueName: \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.193146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.193127    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.194819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.194753    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.197813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.197144    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.267984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.265250    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:54.267984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.265324    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:54.267984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.267519    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc" podStartSLOduration=2.267501374 podStartE2EDuration="2.267501374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:54.265903092 +0000 UTC m=+1383.140173982" watchObservedRunningTime="2026-11-02 20:14:54.267501374 +0000 UTC m=+1383.141772263"
Nov 02 20:14:54.270619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.270587    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkqm\" (UniqueName: \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\") pod \"maas-api-revoke-sub-a70c1ccd-rw62r\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.353597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.353317    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:54.781820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:54.781774    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"]
Nov 02 20:14:55.170708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:55.170678    5099 generic.go:356] "Generic (PLEG): container finished" podID="fa6f89fd-4c0f-461a-9060-129801d6bafd" containerID="9ff485d2cc23871ba06a19d0a451160372c742ba31e090fa00bee9a978981536" exitCode=0
Nov 02 20:14:55.171126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:55.170776    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc" event={"ID":"fa6f89fd-4c0f-461a-9060-129801d6bafd","Type":"ContainerDied","Data":"9ff485d2cc23871ba06a19d0a451160372c742ba31e090fa00bee9a978981536"}
Nov 02 20:14:55.172322 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:55.172297    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r" event={"ID":"850586d8-47f1-45e0-b978-340a6dba4292","Type":"ContainerStarted","Data":"416ba78315adcfd297faa57d5de4a9dd7017b315b211b198fc63bdda1bc17399"}
Nov 02 20:14:55.265052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:55.264952    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:55.265052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:55.265021    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:56.182844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.182734    5099 generic.go:356] "Generic (PLEG): container finished" podID="850586d8-47f1-45e0-b978-340a6dba4292" containerID="2c35a2c9384935415d1c828213a11b86c31a4ef3fba2e2d2393178238c2c8d95" exitCode=0
Nov 02 20:14:56.183290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.182830    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r" event={"ID":"850586d8-47f1-45e0-b978-340a6dba4292","Type":"ContainerDied","Data":"2c35a2c9384935415d1c828213a11b86c31a4ef3fba2e2d2393178238c2c8d95"}
Nov 02 20:14:56.260446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.260349    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:14:56.262891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.262847    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:14:56.375410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.375379    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:56.430926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.430890    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\") pod \"fa6f89fd-4c0f-461a-9060-129801d6bafd\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") "
Nov 02 20:14:56.431121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.430989    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\") pod \"fa6f89fd-4c0f-461a-9060-129801d6bafd\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") "
Nov 02 20:14:56.431121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.431029    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\" (UniqueName: \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\") pod \"fa6f89fd-4c0f-461a-9060-129801d6bafd\" (UID: \"fa6f89fd-4c0f-461a-9060-129801d6bafd\") "
Nov 02 20:14:56.431336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.431309    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca" pod "fa6f89fd-4c0f-461a-9060-129801d6bafd" (UID: "fa6f89fd-4c0f-461a-9060-129801d6bafd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:56.431767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.431743    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:56.433301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.433235    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn" pod "fa6f89fd-4c0f-461a-9060-129801d6bafd" (UID: "fa6f89fd-4c0f-461a-9060-129801d6bafd"). InnerVolumeSpecName "kube-api-access-wtfbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:56.433573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.433554    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca" pod "fa6f89fd-4c0f-461a-9060-129801d6bafd" (UID: "fa6f89fd-4c0f-461a-9060-129801d6bafd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:56.533180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.533135    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6f89fd-4c0f-461a-9060-129801d6bafd-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:56.533180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:56.533176    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfbn\" (UniqueName: \"kubernetes.io/projected/fa6f89fd-4c0f-461a-9060-129801d6bafd-kube-api-access-wtfbn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:57.197593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.197502    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"
Nov 02 20:14:57.197593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.197522    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc" event={"ID":"fa6f89fd-4c0f-461a-9060-129801d6bafd","Type":"ContainerDied","Data":"b6d52820a1efecab3904e182672bd69310799fe62e0ee4529809ab5da94a51e2"}
Nov 02 20:14:57.198098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.197609    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d52820a1efecab3904e182672bd69310799fe62e0ee4529809ab5da94a51e2"
Nov 02 20:14:57.262571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.262268    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:57.263217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.263190    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:57.265049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.265023    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:14:57.385653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.385628    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:57.448946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.448853    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\") pod \"850586d8-47f1-45e0-b978-340a6dba4292\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") "
Nov 02 20:14:57.448946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.448917    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\" (UniqueName: \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\") pod \"850586d8-47f1-45e0-b978-340a6dba4292\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") "
Nov 02 20:14:57.449156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.449071    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\") pod \"850586d8-47f1-45e0-b978-340a6dba4292\" (UID: \"850586d8-47f1-45e0-b978-340a6dba4292\") "
Nov 02 20:14:57.449344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.449316    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca" pod "850586d8-47f1-45e0-b978-340a6dba4292" (UID: "850586d8-47f1-45e0-b978-340a6dba4292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:14:57.449869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.449845    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:57.451170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.451148    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca" pod "850586d8-47f1-45e0-b978-340a6dba4292" (UID: "850586d8-47f1-45e0-b978-340a6dba4292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:14:57.451259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.451180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm" pod "850586d8-47f1-45e0-b978-340a6dba4292" (UID: "850586d8-47f1-45e0-b978-340a6dba4292"). InnerVolumeSpecName "kube-api-access-kqkqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:14:57.551710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.551651    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkqm\" (UniqueName: \"kubernetes.io/projected/850586d8-47f1-45e0-b978-340a6dba4292-kube-api-access-kqkqm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:57.551710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:57.551694    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850586d8-47f1-45e0-b978-340a6dba4292-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:14:58.209333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:58.209293    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"
Nov 02 20:14:58.210333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:58.210235    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r" event={"ID":"850586d8-47f1-45e0-b978-340a6dba4292","Type":"ContainerDied","Data":"416ba78315adcfd297faa57d5de4a9dd7017b315b211b198fc63bdda1bc17399"}
Nov 02 20:14:58.210333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:14:58.210290    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416ba78315adcfd297faa57d5de4a9dd7017b315b211b198fc63bdda1bc17399"
Nov 02 20:15:00.162670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.162580    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:00.163673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163646    5099 cpu_manager.go:414] "containerMap: removing container" podUID="fa6f89fd-4c0f-461a-9060-129801d6bafd" containerName="revoke-keys"
Nov 02 20:15:00.163673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163670    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6f89fd-4c0f-461a-9060-129801d6bafd" containerName="revoke-keys"
Nov 02 20:15:00.163898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163726    5099 cpu_manager.go:414] "containerMap: removing container" podUID="850586d8-47f1-45e0-b978-340a6dba4292" containerName="revoke-keys"
Nov 02 20:15:00.163898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163735    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850586d8-47f1-45e0-b978-340a6dba4292" containerName="revoke-keys"
Nov 02 20:15:00.164004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163936    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6f89fd-4c0f-461a-9060-129801d6bafd" containerName="revoke-keys"
Nov 02 20:15:00.164004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.163958    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="850586d8-47f1-45e0-b978-340a6dba4292" containerName="revoke-keys"
Nov 02 20:15:00.168003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.167983    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.169689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.169655    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:00.175113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.175092    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.177103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.177078    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:00.182553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.182534    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.207116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.207080    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:00.211612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.211591    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.212047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.212022    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:00.216567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.216550    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.217131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.217114    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:15:00.263835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.257376    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:00.263835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.263387    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:00.294751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294546    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7pz\" (UniqueName: \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.294751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294603    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.294751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294691    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nmv\" (UniqueName: \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.294751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294716    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48zf\" (UniqueName: \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.294751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294750    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765km\" (UniqueName: \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.296310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294805    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhps\" (UniqueName: \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.296310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294948    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.296310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.294984    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.296310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.295087    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.296310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.295116    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.300626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.300603    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:00.307214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.307161    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:00.312759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.312735    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:00.396706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.396675    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.396920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.396876    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nmv\" (UniqueName: \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.397022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.396930    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48zf\" (UniqueName: \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.397022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.396968    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765km\" (UniqueName: \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.397022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397010    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhps\" (UniqueName: \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.397369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397321    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.397459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397383    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.397568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397533    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.397630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397591    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.397691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397625    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.397822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.397771    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7pz\" (UniqueName: \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.398133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.398110    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.398133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.398126    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.398352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.398332    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.398394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.398332    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.453010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.452905    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765km\" (UniqueName: \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:00.456827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.453872    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhps\" (UniqueName: \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.456827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.454271    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nmv\" (UniqueName: \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.456827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.454543    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7pz\" (UniqueName: \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.456827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.455536    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48zf\" (UniqueName: \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.495855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.495749    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:00.498767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.497831    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:00.506061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.505991    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:00.569573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.547710    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:00.569573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:00.553160    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:01.215545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.215519    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:01.289068 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:01.289035    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccb2772_582b_4bf2_bfb7_28f40717ca16.slice/crio-bebaca1488258b3ba2b6c7c9f910b33d91671526707974379607c5a7c49c1f6c WatchSource:0}: Error finding container bebaca1488258b3ba2b6c7c9f910b33d91671526707974379607c5a7c49c1f6c: Status 404 returned error can't find the container with id bebaca1488258b3ba2b6c7c9f910b33d91671526707974379607c5a7c49c1f6c
Nov 02 20:15:01.296674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.296599    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:01.296674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.296651    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:15:01.305261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.305225    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:01.305369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.305269    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerStarted","Data":"c795e58fa1fd0e2c363f5225476b0ab95e6ec1c6a74af945fade51ca4292e008"}
Nov 02 20:15:01.305423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.305370    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.316580 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.316551    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:01.333349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.333146    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:01.352241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.344262    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:15:01.355918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.355891    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a2b4e70c-openshift-default-dockercfg-278mb\""
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.476252    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.476410    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.481895    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.482000    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.482089    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.482139    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.490393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.482228    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.516438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.516295    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.516438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.516401    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgmw\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.644773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.644733    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645051    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645130    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645247    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645311    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgmw\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645381    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645466    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645552    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.645606    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.646096    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.646343    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.649843    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.656677    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.657999    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.658253    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.668803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.658468    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.747814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.747727    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgmw\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.766618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.766543    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\") pod \"e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") " pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:01.946354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:01.946077    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:02.282097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.282000    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerStarted","Data":"dcae46b57fb42bfad1681a07ee6a458e9f7589418e8a3d039a4564f28f9aa84a"}
Nov 02 20:15:02.283981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.283949    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerStarted","Data":"a242d6c81a35b99694f3a541344e1648064ff2f7449effbb83a1600b053a993b"}
Nov 02 20:15:02.284136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.284001    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerStarted","Data":"87c2bb28943082154dc283d2bd1dc5dbbde6ebcbb0f5cc5db02ec5b77cae5f63"}
Nov 02 20:15:02.286052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.286026    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerStarted","Data":"0cca145c88ce97567af0f26884406d0e3a8bc7c6bc64036d7623a225cc62c94b"}
Nov 02 20:15:02.286186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.286072    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerStarted","Data":"c0e0d5a530b98f9b451334feb0f978468497f7fecffd0cae43305c126ab0329d"}
Nov 02 20:15:02.287536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.287509    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerStarted","Data":"c89ec4897db9e94b7fac97a89d44a9563d7ec327d0452cd5dc0c5c282f916bb1"}
Nov 02 20:15:02.289327 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.289306    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerStarted","Data":"6fed115515bbdadcb120abed763380804d119314586cd972c5a2cc9a28b80e30"}
Nov 02 20:15:02.289435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.289343    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerStarted","Data":"bebaca1488258b3ba2b6c7c9f910b33d91671526707974379607c5a7c49c1f6c"}
Nov 02 20:15:02.391705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.391653    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" podStartSLOduration=2.391639406 podStartE2EDuration="2.391639406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.388869068 +0000 UTC m=+1391.263139959" watchObservedRunningTime="2026-11-02 20:15:02.391639406 +0000 UTC m=+1391.265910287"
Nov 02 20:15:02.494903 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:02.493332    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dbabb9_86de_41ad_887e_735a9f33d377.slice/crio-6eb50a1afce84540812ee3afbd099902746c8afaff83f9f2f74292a4193ac331 WatchSource:0}: Error finding container 6eb50a1afce84540812ee3afbd099902746c8afaff83f9f2f74292a4193ac331: Status 404 returned error can't find the container with id 6eb50a1afce84540812ee3afbd099902746c8afaff83f9f2f74292a4193ac331
Nov 02 20:15:02.498490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.498412    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:02.498628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.498510    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:02.498628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.498561    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:02.500064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.500045    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:15:02.506447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.506397    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" podStartSLOduration=2.506382832 podStartE2EDuration="2.506382832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.501011575 +0000 UTC m=+1391.375282465" watchObservedRunningTime="2026-11-02 20:15:02.506382832 +0000 UTC m=+1391.380653719"
Nov 02 20:15:02.651812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.651036    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" podStartSLOduration=2.651013569 podStartE2EDuration="2.651013569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.617652416 +0000 UTC m=+1391.491923304" watchObservedRunningTime="2026-11-02 20:15:02.651013569 +0000 UTC m=+1391.525284457"
Nov 02 20:15:02.749810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:02.749654    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" podStartSLOduration=2.749634001 podStartE2EDuration="2.749634001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.735432248 +0000 UTC m=+1391.609703137" watchObservedRunningTime="2026-11-02 20:15:02.749634001 +0000 UTC m=+1391.623904890"
Nov 02 20:15:03.299331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.299298    5099 generic.go:356] "Generic (PLEG): container finished" podID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerID="dcae46b57fb42bfad1681a07ee6a458e9f7589418e8a3d039a4564f28f9aa84a" exitCode=60
Nov 02 20:15:03.299751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.299388    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerDied","Data":"dcae46b57fb42bfad1681a07ee6a458e9f7589418e8a3d039a4564f28f9aa84a"}
Nov 02 20:15:03.299904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.299830    5099 scope.go:122] "RemoveContainer" containerID="dcae46b57fb42bfad1681a07ee6a458e9f7589418e8a3d039a4564f28f9aa84a"
Nov 02 20:15:03.301596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.301519    5099 generic.go:356] "Generic (PLEG): container finished" podID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerID="a242d6c81a35b99694f3a541344e1648064ff2f7449effbb83a1600b053a993b" exitCode=60
Nov 02 20:15:03.301673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.301598    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerDied","Data":"a242d6c81a35b99694f3a541344e1648064ff2f7449effbb83a1600b053a993b"}
Nov 02 20:15:03.301870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.301848    5099 scope.go:122] "RemoveContainer" containerID="a242d6c81a35b99694f3a541344e1648064ff2f7449effbb83a1600b053a993b"
Nov 02 20:15:03.303455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.303433    5099 generic.go:356] "Generic (PLEG): container finished" podID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerID="0cca145c88ce97567af0f26884406d0e3a8bc7c6bc64036d7623a225cc62c94b" exitCode=60
Nov 02 20:15:03.303601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.303515    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerDied","Data":"0cca145c88ce97567af0f26884406d0e3a8bc7c6bc64036d7623a225cc62c94b"}
Nov 02 20:15:03.303696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.303680    5099 scope.go:122] "RemoveContainer" containerID="0cca145c88ce97567af0f26884406d0e3a8bc7c6bc64036d7623a225cc62c94b"
Nov 02 20:15:03.305512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.305495    5099 generic.go:356] "Generic (PLEG): container finished" podID="92d5adc0-7253-492b-beb7-f7be76c12234" containerID="12d85fe74754218f21da8fe8f4bc88fa8437a74c2c00a379202d25b34ab9c071" exitCode=60
Nov 02 20:15:03.305590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.305555    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerDied","Data":"12d85fe74754218f21da8fe8f4bc88fa8437a74c2c00a379202d25b34ab9c071"}
Nov 02 20:15:03.305770 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.305753    5099 scope.go:122] "RemoveContainer" containerID="12d85fe74754218f21da8fe8f4bc88fa8437a74c2c00a379202d25b34ab9c071"
Nov 02 20:15:03.307367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.307346    5099 generic.go:356] "Generic (PLEG): container finished" podID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerID="6fed115515bbdadcb120abed763380804d119314586cd972c5a2cc9a28b80e30" exitCode=60
Nov 02 20:15:03.307475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.307417    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerDied","Data":"6fed115515bbdadcb120abed763380804d119314586cd972c5a2cc9a28b80e30"}
Nov 02 20:15:03.307594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.307580    5099 scope.go:122] "RemoveContainer" containerID="6fed115515bbdadcb120abed763380804d119314586cd972c5a2cc9a28b80e30"
Nov 02 20:15:03.309443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.309419    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerStarted","Data":"d6e07fc020ab671ae0dc955862d82c3d3292f70f4768bace06581231276ae371"}
Nov 02 20:15:03.309533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.309467    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerStarted","Data":"6eb50a1afce84540812ee3afbd099902746c8afaff83f9f2f74292a4193ac331"}
Nov 02 20:15:03.626055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.625989    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podStartSLOduration=2.625968406 podStartE2EDuration="2.625968406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.622862669 +0000 UTC m=+1392.497133554" watchObservedRunningTime="2026-11-02 20:15:03.625968406 +0000 UTC m=+1392.500239298"
Nov 02 20:15:03.947376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.947290    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:03.952495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:03.952468    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:04.320937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.320839    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerStarted","Data":"1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"}
Nov 02 20:15:04.323289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.323264    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerStarted","Data":"8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"}
Nov 02 20:15:04.325271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.325247    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerStarted","Data":"f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"}
Nov 02 20:15:04.327012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.326989    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerStarted","Data":"63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"}
Nov 02 20:15:04.328973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.328951    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerStarted","Data":"73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"}
Nov 02 20:15:04.329057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.328995    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:04.329975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.329956    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:15:04.776276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:04.776226    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" podStartSLOduration=4.776212357 podStartE2EDuration="4.776212357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.771626733 +0000 UTC m=+1393.645897614" watchObservedRunningTime="2026-11-02 20:15:04.776212357 +0000 UTC m=+1393.650483253"
Nov 02 20:15:05.342332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.342297    5099 generic.go:356] "Generic (PLEG): container finished" podID="92d5adc0-7253-492b-beb7-f7be76c12234" containerID="73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a" exitCode=60
Nov 02 20:15:05.342992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.342354    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerDied","Data":"73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"}
Nov 02 20:15:05.342992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.342422    5099 scope.go:122] "RemoveContainer" containerID="12d85fe74754218f21da8fe8f4bc88fa8437a74c2c00a379202d25b34ab9c071"
Nov 02 20:15:05.343302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.343279    5099 scope.go:122] "RemoveContainer" containerID="73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"
Nov 02 20:15:05.343754 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:05.343725    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst_odh-ai-gateway-infra(92d5adc0-7253-492b-beb7-f7be76c12234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" podUID="92d5adc0-7253-492b-beb7-f7be76c12234"
Nov 02 20:15:05.346637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.346609    5099 generic.go:356] "Generic (PLEG): container finished" podID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerID="1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422" exitCode=60
Nov 02 20:15:05.347081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.347062    5099 scope.go:122] "RemoveContainer" containerID="1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"
Nov 02 20:15:05.347193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.347060    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerDied","Data":"1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"}
Nov 02 20:15:05.364841 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:05.364315    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6_odh-ai-gateway-infra(dccb2772-582b-4bf2-bfb7-28f40717ca16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16"
Nov 02 20:15:05.387810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.384069    5099 generic.go:356] "Generic (PLEG): container finished" podID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerID="8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8" exitCode=60
Nov 02 20:15:05.387810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.384260    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerDied","Data":"8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"}
Nov 02 20:15:05.387810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.384822    5099 scope.go:122] "RemoveContainer" containerID="8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"
Nov 02 20:15:05.387810 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:05.385275    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz_odh-ai-gateway-infra(460dbbeb-c1e9-4881-86ae-652f6c08620d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d"
Nov 02 20:15:05.523804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.518388    5099 generic.go:356] "Generic (PLEG): container finished" podID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerID="f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115" exitCode=60
Nov 02 20:15:05.523804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.518497    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerDied","Data":"f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"}
Nov 02 20:15:05.523804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.521107    5099 scope.go:122] "RemoveContainer" containerID="6fed115515bbdadcb120abed763380804d119314586cd972c5a2cc9a28b80e30"
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.531945    5099 generic.go:356] "Generic (PLEG): container finished" podID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerID="63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec" exitCode=60
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.533110    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerDied","Data":"63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"}
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.534456    5099 scope.go:122] "RemoveContainer" containerID="f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.534652    5099 scope.go:122] "RemoveContainer" containerID="63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:05.535052    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8_odh-ai-gateway-infra(314a41ee-30ca-4db3-b3aa-c6d549ee449d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d"
Nov 02 20:15:05.538978 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:05.535688    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t_odh-ai-gateway-infra(e7997b70-ff6f-48b9-902c-3eb22e949c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07"
Nov 02 20:15:05.588878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.580981    5099 scope.go:122] "RemoveContainer" containerID="dcae46b57fb42bfad1681a07ee6a458e9f7589418e8a3d039a4564f28f9aa84a"
Nov 02 20:15:05.608027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.608001    5099 scope.go:122] "RemoveContainer" containerID="a242d6c81a35b99694f3a541344e1648064ff2f7449effbb83a1600b053a993b"
Nov 02 20:15:05.625360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:05.625333    5099 scope.go:122] "RemoveContainer" containerID="0cca145c88ce97567af0f26884406d0e3a8bc7c6bc64036d7623a225cc62c94b"
Nov 02 20:15:06.544224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:06.544192    5099 scope.go:122] "RemoveContainer" containerID="8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"
Nov 02 20:15:06.544740 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:06.544626    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz_odh-ai-gateway-infra(460dbbeb-c1e9-4881-86ae-652f6c08620d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d"
Nov 02 20:15:06.546143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:06.546120    5099 scope.go:122] "RemoveContainer" containerID="f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"
Nov 02 20:15:06.546525 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:06.546497    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t_odh-ai-gateway-infra(e7997b70-ff6f-48b9-902c-3eb22e949c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07"
Nov 02 20:15:06.547987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:06.547969    5099 scope.go:122] "RemoveContainer" containerID="63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"
Nov 02 20:15:06.548252 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:06.548229    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8_odh-ai-gateway-infra(314a41ee-30ca-4db3-b3aa-c6d549ee449d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d"
Nov 02 20:15:06.549903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:06.549875    5099 scope.go:122] "RemoveContainer" containerID="73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"
Nov 02 20:15:06.550131 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:06.550113    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst_odh-ai-gateway-infra(92d5adc0-7253-492b-beb7-f7be76c12234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" podUID="92d5adc0-7253-492b-beb7-f7be76c12234"
Nov 02 20:15:06.551708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:06.551689    5099 scope.go:122] "RemoveContainer" containerID="1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"
Nov 02 20:15:06.551941 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:06.551918    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6_odh-ai-gateway-infra(dccb2772-582b-4bf2-bfb7-28f40717ca16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16"
Nov 02 20:15:10.086809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:10.064958    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:10.086809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:10.070887    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:10.091039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:10.090982    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:10.138808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:10.131466    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:10.658062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:10.654368    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.162810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.120199    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.350911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.122869    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.351174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.259597    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.351296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.312973    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.351397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.313101    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.351498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.326005    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.351594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.341906    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:11.764807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:11.753797    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.191888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.191851    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:15:12.192468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.192446    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:15:12.192604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.192592    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:15:12.192715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.192704    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:15:12.192869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.192854    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.476310    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.492090    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.492442    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.492717    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.493028    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.493296    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.493981    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:15:12.499076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.494046    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:15:12.784903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:12.784807    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.479818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.478500    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.502563    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.502862    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.503055    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.503211    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.503364    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.522913    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:13.559948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.522980    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:13.870148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.870102    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Nov 02 20:15:13.870463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.870435    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Nov 02 20:15:13.870621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:13.870603    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.503944    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.504380    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.504682    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.504977    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.505229    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:14.506809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:14.505494    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:15.243868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:15.224930    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:15.243868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:15.225015    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:15.278359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:15.278297    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:15:16.451809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.439838    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:16.678543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.678492    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:16.974493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.699966    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:17.009810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.700049    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:17.036874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.700090    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:17.038574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.700152    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:17.039344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.700570    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:17.052853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:16.847291    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:17.053155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.053118    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:17.190806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.165471    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:17.250861 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:17.224494    5099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.901s"
Nov 02 20:15:17.250861 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:17.242083    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:15:17.250861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.248071    5099 scope.go:122] "RemoveContainer" containerID="73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"
Nov 02 20:15:17.251707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.251506    5099 scope.go:122] "RemoveContainer" containerID="8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"
Nov 02 20:15:17.837875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.814579    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:18.243908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.975961    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:18.243908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:17.977927    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:18.243908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.066869    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:18.278965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.185173    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:18.278965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.268528    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:18.317368 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.317323    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:18.317630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.317599    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:18.407810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.382312    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:18.437135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.437033    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:18.494198 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:18.494159    5099 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.219s"
Nov 02 20:15:18.494520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.494501    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:15:18.870884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.870851    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:15:18.936150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.936061    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:18.937208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.937177    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:18.980671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:18.980581    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:19.118830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.068917    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:19.118830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.099608    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 20:15:19.226806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.222960    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:15:19.226806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.224023    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:19.245093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.231866    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:19.246062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.246005    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:19.311805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.307535    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:19.492394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.492362    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.523041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.522988    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:19.548809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.544883    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2d73d56c-openshift-default-dockercfg-9wkfn\""
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709008    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709157    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709261    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709296    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709325    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709517    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709557    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdkw\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709594    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.757860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.709638    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.813832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.804952    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:15:19.824195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.824167    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:15:19.824581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.824562    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.811541    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdkw\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.836923    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.836976    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837065    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837200    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837320    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837342    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837849    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.837909    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.852489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.838147    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.890724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.889419    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.890724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.889710    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:19.911920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.911890    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.044469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.918759    5099 scope.go:122] "RemoveContainer" containerID="f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"
Nov 02 20:15:20.051043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.921164    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.051289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:19.989847    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:20.131890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.124950    5099 scope.go:122] "RemoveContainer" containerID="1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"
Nov 02 20:15:20.132906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.132880    5099 scope.go:122] "RemoveContainer" containerID="63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"
Nov 02 20:15:20.172160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.172132    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:15:20.172574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.172467    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.208822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.203638    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:15:20.209644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.209235    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdkw\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.213875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.213759    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.214321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.214283    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55475\" (UniqueName: \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.237866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.236588    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.265055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.260599    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2b4e70c\""
Nov 02 20:15:20.319404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.319292    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:20.320038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.320000    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:20.735294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.735243    5099 generic.go:356] "Generic (PLEG): container finished" podID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerID="7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9" exitCode=60
Nov 02 20:15:20.735551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.446195    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2b4e70c-dockercfg-dldhr\""
Nov 02 20:15:20.735706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.735686    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerDied","Data":"7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9"}
Nov 02 20:15:20.735918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.735890    5099 scope.go:122] "RemoveContainer" containerID="8195d9d5e6d85b3dcf673626070210cc5e21b61e847c4d2a78e36756b073dcc8"
Nov 02 20:15:20.736131 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:20.555318    5099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460dbbeb_c1e9_4881_86ae_652f6c08620d.slice/crio-conmon-7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 20:15:20.753550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.381955    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:20.753550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.611579    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.753550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.718517    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55475\" (UniqueName: \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.756520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.753986    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.756520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.755302    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.776551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.775172    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\") pod \"e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") " pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.779733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.525339    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:20.784838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.540396    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 20:15:20.784838 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:20.569203    5099 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460dbbeb_c1e9_4881_86ae_652f6c08620d.slice/crio-conmon-7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 20:15:20.789906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.787052    5099 scope.go:122] "RemoveContainer" containerID="7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9"
Nov 02 20:15:20.801964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.793345    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:15:20.819941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.807109    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:15:20.819941 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:20.811022    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz_odh-ai-gateway-infra(460dbbeb-c1e9-4881-86ae-652f6c08620d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d"
Nov 02 20:15:20.819941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.818459    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:20.820225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.820087    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:15:20.832831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.832736    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:15:20.833272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.833182    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55475\" (UniqueName: \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\") pod \"payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:20.859194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:20.859163    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:15:21.109812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.109765    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:21.416019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.412680    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:21.416019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.412777    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:21.416019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.413304    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:21.423907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.422026    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:15:21.549291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.549259    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:15:21.551359 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:21.551330    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44432bb6_6c06_4654_85a2_fbc34db18034.slice/crio-e9f6b505f2dd3bb407afa96e8e8863c528d7ca77d5085a1a4c876cb93259fac0 WatchSource:0}: Error finding container e9f6b505f2dd3bb407afa96e8e8863c528d7ca77d5085a1a4c876cb93259fac0: Status 404 returned error can't find the container with id e9f6b505f2dd3bb407afa96e8e8863c528d7ca77d5085a1a4c876cb93259fac0
Nov 02 20:15:21.756083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.756041    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerStarted","Data":"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"}
Nov 02 20:15:21.756277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.756104    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerStarted","Data":"57a2adfbad12155a3c054c2829b19176510d77fa0865dd150f89c905fe4fc800"}
Nov 02 20:15:21.760609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.760583    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerStarted","Data":"4f03eaf1c942117262d0fd00158a0b9da561b578a0193a9da99831860fd857d9"}
Nov 02 20:15:21.763396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.763106    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerStarted","Data":"cc0fd43ab3d57f392c696cd8706a9309a222623229bf432d5fbe6a7e171c0aae"}
Nov 02 20:15:21.765714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.765684    5099 generic.go:356] "Generic (PLEG): container finished" podID="92d5adc0-7253-492b-beb7-f7be76c12234" containerID="6dd3f431556230d4667860322705714d68b314f6adabfce511b44d28b9e6d96e" exitCode=60
Nov 02 20:15:21.765878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.765745    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerDied","Data":"6dd3f431556230d4667860322705714d68b314f6adabfce511b44d28b9e6d96e"}
Nov 02 20:15:21.765878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.765826    5099 scope.go:122] "RemoveContainer" containerID="73f027f69f37a56e6c9a8ac3678dea1b396f45ef96f96d8807fd19e05ca5428a"
Nov 02 20:15:21.766123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.766107    5099 scope.go:122] "RemoveContainer" containerID="6dd3f431556230d4667860322705714d68b314f6adabfce511b44d28b9e6d96e"
Nov 02 20:15:21.766369 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:21.766350    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst_odh-ai-gateway-infra(92d5adc0-7253-492b-beb7-f7be76c12234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" podUID="92d5adc0-7253-492b-beb7-f7be76c12234"
Nov 02 20:15:21.770732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.770711    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerStarted","Data":"21f06f5b8ae9ddadde50bbffffbc680231241aacdf29b412453137b73bfb1cd3"}
Nov 02 20:15:21.772999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.772971    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" event={"ID":"44432bb6-6c06-4654-85a2-fbc34db18034","Type":"ContainerStarted","Data":"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"}
Nov 02 20:15:21.773112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.773028    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:21.773112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.773054    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" event={"ID":"44432bb6-6c06-4654-85a2-fbc34db18034","Type":"ContainerStarted","Data":"e9f6b505f2dd3bb407afa96e8e8863c528d7ca77d5085a1a4c876cb93259fac0"}
Nov 02 20:15:21.820777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.820748    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:21.876237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:21.876161    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podStartSLOduration=4.876135355 podStartE2EDuration="4.876135355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:21.868799551 +0000 UTC m=+1410.743070434" watchObservedRunningTime="2026-11-02 20:15:21.876135355 +0000 UTC m=+1410.750406245"
Nov 02 20:15:22.221805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.220908    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" podStartSLOduration=4.22088731 podStartE2EDuration="4.22088731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:22.097834172 +0000 UTC m=+1410.972105060" watchObservedRunningTime="2026-11-02 20:15:22.22088731 +0000 UTC m=+1411.095158200"
Nov 02 20:15:22.512801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.512232    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:22.702921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.702897    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:22.785609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.785578    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"
Nov 02 20:15:22.785609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.785599    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz" event={"ID":"460dbbeb-c1e9-4881-86ae-652f6c08620d","Type":"ContainerDied","Data":"c795e58fa1fd0e2c363f5225476b0ab95e6ec1c6a74af945fade51ca4292e008"}
Nov 02 20:15:22.785881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.785669    5099 scope.go:122] "RemoveContainer" containerID="7faba98c3f3403762e79b3e891e16202d95744c2088009be64e77d5f4f1f3cd9"
Nov 02 20:15:22.787902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.787873    5099 generic.go:356] "Generic (PLEG): container finished" podID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerID="4f03eaf1c942117262d0fd00158a0b9da561b578a0193a9da99831860fd857d9" exitCode=60
Nov 02 20:15:22.788044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.787939    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerDied","Data":"4f03eaf1c942117262d0fd00158a0b9da561b578a0193a9da99831860fd857d9"}
Nov 02 20:15:22.788346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.788330    5099 scope.go:122] "RemoveContainer" containerID="4f03eaf1c942117262d0fd00158a0b9da561b578a0193a9da99831860fd857d9"
Nov 02 20:15:22.788735 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:22.788703    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t_odh-ai-gateway-infra(e7997b70-ff6f-48b9-902c-3eb22e949c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07"
Nov 02 20:15:22.790015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.789997    5099 generic.go:356] "Generic (PLEG): container finished" podID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerID="cc0fd43ab3d57f392c696cd8706a9309a222623229bf432d5fbe6a7e171c0aae" exitCode=60
Nov 02 20:15:22.790108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.790074    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerDied","Data":"cc0fd43ab3d57f392c696cd8706a9309a222623229bf432d5fbe6a7e171c0aae"}
Nov 02 20:15:22.790399 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.790384    5099 scope.go:122] "RemoveContainer" containerID="cc0fd43ab3d57f392c696cd8706a9309a222623229bf432d5fbe6a7e171c0aae"
Nov 02 20:15:22.790710 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:22.790690    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8_odh-ai-gateway-infra(314a41ee-30ca-4db3-b3aa-c6d549ee449d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d"
Nov 02 20:15:22.793569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.793550    5099 generic.go:356] "Generic (PLEG): container finished" podID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerID="21f06f5b8ae9ddadde50bbffffbc680231241aacdf29b412453137b73bfb1cd3" exitCode=60
Nov 02 20:15:22.793652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.793621    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerDied","Data":"21f06f5b8ae9ddadde50bbffffbc680231241aacdf29b412453137b73bfb1cd3"}
Nov 02 20:15:22.794215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.794197    5099 scope.go:122] "RemoveContainer" containerID="21f06f5b8ae9ddadde50bbffffbc680231241aacdf29b412453137b73bfb1cd3"
Nov 02 20:15:22.794432 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:22.794412    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6_odh-ai-gateway-infra(dccb2772-582b-4bf2-bfb7-28f40717ca16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16"
Nov 02 20:15:22.797362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.797344    5099 scope.go:122] "RemoveContainer" containerID="f5a750f0d665cdada1a1e2936a7ba9f8e6dfa8400e4a77297fa3d30647fbc115"
Nov 02 20:15:22.812603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.812579    5099 scope.go:122] "RemoveContainer" containerID="63869c436c9e1617916f64c36f76348d9ab98cc4048ea144d1002d648cd919ec"
Nov 02 20:15:22.827236 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.827209    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:22.832293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.832254    5099 scope.go:122] "RemoveContainer" containerID="1373601382df6a3431d7624a760c7c3be8a3e47bac896fa014bd298c56218422"
Nov 02 20:15:22.859235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.859205    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\") pod \"460dbbeb-c1e9-4881-86ae-652f6c08620d\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") "
Nov 02 20:15:22.859509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.859487    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\" (UniqueName: \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\") pod \"460dbbeb-c1e9-4881-86ae-652f6c08620d\" (UID: \"460dbbeb-c1e9-4881-86ae-652f6c08620d\") "
Nov 02 20:15:22.860001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.859890    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca" pod "460dbbeb-c1e9-4881-86ae-652f6c08620d" (UID: "460dbbeb-c1e9-4881-86ae-652f6c08620d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:22.860392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.860367    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/460dbbeb-c1e9-4881-86ae-652f6c08620d-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.862139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.862108    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz" pod "460dbbeb-c1e9-4881-86ae-652f6c08620d" (UID: "460dbbeb-c1e9-4881-86ae-652f6c08620d"). InnerVolumeSpecName "kube-api-access-mz7pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:22.962912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.962833    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7pz\" (UniqueName: \"kubernetes.io/projected/460dbbeb-c1e9-4881-86ae-652f6c08620d-kube-api-access-mz7pz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:22.987891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:22.987859    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:23.227933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.227839    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:23.345757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.342813    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:23.420131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.420102    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:23.427022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.426992    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894175-5dsqz"]
Nov 02 20:15:23.472497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.472463    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:23.563292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.563263    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:23.713338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.713293    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\") pod \"92d5adc0-7253-492b-beb7-f7be76c12234\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") "
Nov 02 20:15:23.713509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.713493    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\" (UniqueName: \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\") pod \"92d5adc0-7253-492b-beb7-f7be76c12234\" (UID: \"92d5adc0-7253-492b-beb7-f7be76c12234\") "
Nov 02 20:15:23.737620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.734925    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca" pod "92d5adc0-7253-492b-beb7-f7be76c12234" (UID: "92d5adc0-7253-492b-beb7-f7be76c12234"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:23.770907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.764987    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv" pod "92d5adc0-7253-492b-beb7-f7be76c12234" (UID: "92d5adc0-7253-492b-beb7-f7be76c12234"). InnerVolumeSpecName "kube-api-access-f5nmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:23.820357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.816140    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d5adc0-7253-492b-beb7-f7be76c12234-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:23.820357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.816182    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nmv\" (UniqueName: \"kubernetes.io/projected/92d5adc0-7253-492b-beb7-f7be76c12234-kube-api-access-f5nmv\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:23.852860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.852268    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst" event={"ID":"92d5adc0-7253-492b-beb7-f7be76c12234","Type":"ContainerDied","Data":"c89ec4897db9e94b7fac97a89d44a9563d7ec327d0452cd5dc0c5c282f916bb1"}
Nov 02 20:15:23.852860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.852341    5099 scope.go:122] "RemoveContainer" containerID="6dd3f431556230d4667860322705714d68b314f6adabfce511b44d28b9e6d96e"
Nov 02 20:15:23.852860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.852462    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"
Nov 02 20:15:23.873231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.873155    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:23.874419 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:23.874186    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:15:24.110515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.110483    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:24.120709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.120683    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\" (UniqueName: \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\") pod \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") "
Nov 02 20:15:24.120874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.120737    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\") pod \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\" (UID: \"314a41ee-30ca-4db3-b3aa-c6d549ee449d\") "
Nov 02 20:15:24.121154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.121126    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca" pod "314a41ee-30ca-4db3-b3aa-c6d549ee449d" (UID: "314a41ee-30ca-4db3-b3aa-c6d549ee449d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:24.121508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.121485    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314a41ee-30ca-4db3-b3aa-c6d549ee449d-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.123820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.123768    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf" pod "314a41ee-30ca-4db3-b3aa-c6d549ee449d" (UID: "314a41ee-30ca-4db3-b3aa-c6d549ee449d"). InnerVolumeSpecName "kube-api-access-n48zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:24.179292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.179266    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:24.186134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.186106    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:24.222442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.222306    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\") pod \"dccb2772-582b-4bf2-bfb7-28f40717ca16\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") "
Nov 02 20:15:24.222639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.222626    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\" (UniqueName: \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\") pod \"dccb2772-582b-4bf2-bfb7-28f40717ca16\" (UID: \"dccb2772-582b-4bf2-bfb7-28f40717ca16\") "
Nov 02 20:15:24.222689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.222681    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\" (UniqueName: \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\") pod \"e7997b70-ff6f-48b9-902c-3eb22e949c07\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") "
Nov 02 20:15:24.222776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.222701    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\") pod \"e7997b70-ff6f-48b9-902c-3eb22e949c07\" (UID: \"e7997b70-ff6f-48b9-902c-3eb22e949c07\") "
Nov 02 20:15:24.222899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.222842    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca" pod "dccb2772-582b-4bf2-bfb7-28f40717ca16" (UID: "dccb2772-582b-4bf2-bfb7-28f40717ca16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:24.223251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.223226    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca" pod "e7997b70-ff6f-48b9-902c-3eb22e949c07" (UID: "e7997b70-ff6f-48b9-902c-3eb22e949c07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:24.223824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.223770    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7997b70-ff6f-48b9-902c-3eb22e949c07-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.223824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.223822    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n48zf\" (UniqueName: \"kubernetes.io/projected/314a41ee-30ca-4db3-b3aa-c6d549ee449d-kube-api-access-n48zf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.224001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.223838    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dccb2772-582b-4bf2-bfb7-28f40717ca16-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.224995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.224968    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps" pod "dccb2772-582b-4bf2-bfb7-28f40717ca16" (UID: "dccb2772-582b-4bf2-bfb7-28f40717ca16"). InnerVolumeSpecName "kube-api-access-ljhps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:24.225132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.225077    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km" pod "e7997b70-ff6f-48b9-902c-3eb22e949c07" (UID: "e7997b70-ff6f-48b9-902c-3eb22e949c07"). InnerVolumeSpecName "kube-api-access-765km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:24.325425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.325340    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljhps\" (UniqueName: \"kubernetes.io/projected/dccb2772-582b-4bf2-bfb7-28f40717ca16-kube-api-access-ljhps\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.325425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.325367    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765km\" (UniqueName: \"kubernetes.io/projected/e7997b70-ff6f-48b9-902c-3eb22e949c07-kube-api-access-765km\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:24.446538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.446500    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:24.459422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.459313    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894175-jxpst"]
Nov 02 20:15:24.881872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.881712    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"
Nov 02 20:15:24.881872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.881721    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t" event={"ID":"e7997b70-ff6f-48b9-902c-3eb22e949c07","Type":"ContainerDied","Data":"87c2bb28943082154dc283d2bd1dc5dbbde6ebcbb0f5cc5db02ec5b77cae5f63"}
Nov 02 20:15:24.881872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.881869    5099 scope.go:122] "RemoveContainer" containerID="4f03eaf1c942117262d0fd00158a0b9da561b578a0193a9da99831860fd857d9"
Nov 02 20:15:24.883760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.883726    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8" event={"ID":"314a41ee-30ca-4db3-b3aa-c6d549ee449d","Type":"ContainerDied","Data":"c0e0d5a530b98f9b451334feb0f978468497f7fecffd0cae43305c126ab0329d"}
Nov 02 20:15:24.883760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.883739    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"
Nov 02 20:15:24.886793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.886766    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"
Nov 02 20:15:24.887023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.886803    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6" event={"ID":"dccb2772-582b-4bf2-bfb7-28f40717ca16","Type":"ContainerDied","Data":"bebaca1488258b3ba2b6c7c9f910b33d91671526707974379607c5a7c49c1f6c"}
Nov 02 20:15:24.898952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.898923    5099 scope.go:122] "RemoveContainer" containerID="cc0fd43ab3d57f392c696cd8706a9309a222623229bf432d5fbe6a7e171c0aae"
Nov 02 20:15:24.908676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:24.908653    5099 scope.go:122] "RemoveContainer" containerID="21f06f5b8ae9ddadde50bbffffbc680231241aacdf29b412453137b73bfb1cd3"
Nov 02 20:15:25.039644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.039612    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:25.046931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.046900    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894175-kgtx8"]
Nov 02 20:15:25.152745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.152648    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:25.160931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.160903    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894175-xrq9t"]
Nov 02 20:15:25.262996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.262964    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" path="/var/lib/kubelet/pods/314a41ee-30ca-4db3-b3aa-c6d549ee449d/volumes"
Nov 02 20:15:25.263496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.263481    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" path="/var/lib/kubelet/pods/460dbbeb-c1e9-4881-86ae-652f6c08620d/volumes"
Nov 02 20:15:25.264011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.263987    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" path="/var/lib/kubelet/pods/92d5adc0-7253-492b-beb7-f7be76c12234/volumes"
Nov 02 20:15:25.264847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.264775    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" path="/var/lib/kubelet/pods/e7997b70-ff6f-48b9-902c-3eb22e949c07/volumes"
Nov 02 20:15:25.265163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.265148    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:25.265231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:25.265179    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894175-szdn6"]
Nov 02 20:15:27.267193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.267159    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" path="/var/lib/kubelet/pods/dccb2772-582b-4bf2-bfb7-28f40717ca16/volumes"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.543515    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544883    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544907    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544920    5099 cpu_manager.go:414] "containerMap: removing container" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544927    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544970    5099 cpu_manager.go:414] "containerMap: removing container" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544979    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.544993    5099 cpu_manager.go:414] "containerMap: removing container" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545001    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545030    5099 cpu_manager.go:414] "containerMap: removing container" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545037    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545052    5099 cpu_manager.go:414] "containerMap: removing container" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545059    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545106    5099 cpu_manager.go:414] "containerMap: removing container" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545114    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545127    5099 cpu_manager.go:414] "containerMap: removing container" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545135    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545163    5099 cpu_manager.go:414] "containerMap: removing container" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545181    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545192    5099 cpu_manager.go:414] "containerMap: removing container" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545201    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545222    5099 cpu_manager.go:414] "containerMap: removing container" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545229    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545241    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545273    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545554    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545581    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545592    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545602    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545618    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545675    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545692    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="460dbbeb-c1e9-4881-86ae-652f6c08620d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545713    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545755    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545767    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545805    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.551021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.545817    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:27.560804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.555853    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.590076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.590035    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.597060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.597024    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6f4\" (UniqueName: \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.598008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.597865    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:15:27.638009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.635108    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2d73d56c-dockercfg-bcljf\""
Nov 02 20:15:27.638009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.635233    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2d73d56c\""
Nov 02 20:15:27.767816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.761628    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6f4\" (UniqueName: \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.768455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.768421    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.769537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.769511    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.839701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.838035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6f4\" (UniqueName: \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\") pod \"payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:27.928673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:27.928647    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:28.428134 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:28.428094    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77819e2b_a43a_45d0_b128_81133cd8b063.slice/crio-e70afc0f34712f47163a0e390d3b522623b9c78b80a1393c1b31f7242a8dcc19 WatchSource:0}: Error finding container e70afc0f34712f47163a0e390d3b522623b9c78b80a1393c1b31f7242a8dcc19: Status 404 returned error can't find the container with id e70afc0f34712f47163a0e390d3b522623b9c78b80a1393c1b31f7242a8dcc19
Nov 02 20:15:28.428593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:28.428563    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:15:28.935319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:28.935279    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" event={"ID":"77819e2b-a43a-45d0-b128-81133cd8b063","Type":"ContainerStarted","Data":"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"}
Nov 02 20:15:28.935493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:28.935350    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" event={"ID":"77819e2b-a43a-45d0-b128-81133cd8b063","Type":"ContainerStarted","Data":"e70afc0f34712f47163a0e390d3b522623b9c78b80a1393c1b31f7242a8dcc19"}
Nov 02 20:15:28.935493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:28.935380    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:29.037532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.037503    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"]
Nov 02 20:15:29.038263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038245    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:29.038330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038264    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:29.038330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038275    5099 cpu_manager.go:414] "containerMap: removing container" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:29.038330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038280    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:29.038330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038303    5099 cpu_manager.go:414] "containerMap: removing container" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:29.038330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038308    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314a41ee-30ca-4db3-b3aa-c6d549ee449d" containerName="cleanup"
Nov 02 20:15:29.038476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038442    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7997b70-ff6f-48b9-902c-3eb22e949c07" containerName="cleanup"
Nov 02 20:15:29.038476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038453    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d5adc0-7253-492b-beb7-f7be76c12234" containerName="cleanup"
Nov 02 20:15:29.038476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.038466    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccb2772-582b-4bf2-bfb7-28f40717ca16" containerName="cleanup"
Nov 02 20:15:29.042530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.042467    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.043632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.043591    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" podStartSLOduration=2.04358045 podStartE2EDuration="2.04358045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:29.042460681 +0000 UTC m=+1417.916731570" watchObservedRunningTime="2026-11-02 20:15:29.04358045 +0000 UTC m=+1417.917851337"
Nov 02 20:15:29.055457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.055434    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"]
Nov 02 20:15:29.092326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.092301    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:15:29.092459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.092342    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:15:29.117161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.117131    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.117381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.117358    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.117467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.117449    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqzt\" (UniqueName: \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.237992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.236338    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.237992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.236539    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.237992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.236603    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqzt\" (UniqueName: \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.260804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.260752    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.279311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.279220    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.351435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.351402    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqzt\" (UniqueName: \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\") pod \"maas-api-revoke-sub-dba4a9a3-2b4p8\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.380856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.364131    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:29.797474 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:29.797441    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c811d7d_97c0_45ef_aa4c_2d47f10e72d9.slice/crio-3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129 WatchSource:0}: Error finding container 3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129: Status 404 returned error can't find the container with id 3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129
Nov 02 20:15:29.798956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.798930    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"]
Nov 02 20:15:29.946607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:29.946567    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8" event={"ID":"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9","Type":"ContainerStarted","Data":"3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129"}
Nov 02 20:15:30.730474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.730420    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:15:30.735659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.735639    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.773857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.773812    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.773863    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.773968    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774105    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774170    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774231    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774317    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvck\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774586    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.774702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.774688    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.832946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.832917    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7f75722a-openshift-default-dockercfg-j65dd\""
Nov 02 20:15:30.841164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.841134    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:15:30.876459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876422    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876463    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876484    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876503    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876565    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876614    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876662    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.876838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876697    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvck\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876967    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.876985    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.877168    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.877235    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.877318    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.877664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.877637    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.879076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.879055    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:30.879546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:30.879528    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:31.003549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.000690    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:31.003549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.002497    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvck\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\") pod \"e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") " pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:31.063808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.062508    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:31.127808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.127497    5099 generic.go:356] "Generic (PLEG): container finished" podID="2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" containerID="388df67fea9590150f00836f7a8cd8601d7796ce5967e3fd1511e689e2aa8b6c" exitCode=0
Nov 02 20:15:31.129447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.129411    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8" event={"ID":"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9","Type":"ContainerDied","Data":"388df67fea9590150f00836f7a8cd8601d7796ce5967e3fd1511e689e2aa8b6c"}
Nov 02 20:15:31.631362 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:31.631332    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dbe970_e0e1_42ba_b4c4_2ebb5d22528f.slice/crio-fca632f0231938e915e6e1f068a9ceb7a4a2b5bb5998a83b8cd6901285b43293 WatchSource:0}: Error finding container fca632f0231938e915e6e1f068a9ceb7a4a2b5bb5998a83b8cd6901285b43293: Status 404 returned error can't find the container with id fca632f0231938e915e6e1f068a9ceb7a4a2b5bb5998a83b8cd6901285b43293
Nov 02 20:15:31.633448 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.633426    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:15:31.634358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.634328    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:31.634414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.634390    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:31.634446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:31.634418    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:15:32.139600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.139562    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerStarted","Data":"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"}
Nov 02 20:15:32.140013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.139640    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerStarted","Data":"fca632f0231938e915e6e1f068a9ceb7a4a2b5bb5998a83b8cd6901285b43293"}
Nov 02 20:15:32.258397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.258342    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podStartSLOduration=2.258325958 podStartE2EDuration="2.258325958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:32.249263577 +0000 UTC m=+1421.123534466" watchObservedRunningTime="2026-11-02 20:15:32.258325958 +0000 UTC m=+1421.132596845"
Nov 02 20:15:32.341414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.341386    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:32.379804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.364598    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\") pod \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") "
Nov 02 20:15:32.379804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.364716    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\") pod \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") "
Nov 02 20:15:32.379804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.364758    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\" (UniqueName: \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\") pod \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\" (UID: \"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9\") "
Nov 02 20:15:32.406935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.399887    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca" pod "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" (UID: "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:15:32.406935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.403324    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca" pod "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" (UID: "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:15:32.430827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.430772    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt" pod "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" (UID: "2c811d7d-97c0-45ef-aa4c-2d47f10e72d9"). InnerVolumeSpecName "kube-api-access-wcqzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:15:32.519805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.500091    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:32.520746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.520714    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:32.520937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.520923    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqzt\" (UniqueName: \"kubernetes.io/projected/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9-kube-api-access-wcqzt\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:15:32.796815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:32.796722    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:15:33.066005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.065937    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:33.072588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.072559    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:33.150115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.150084    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"
Nov 02 20:15:33.150115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.150105    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8" event={"ID":"2c811d7d-97c0-45ef-aa4c-2d47f10e72d9","Type":"ContainerDied","Data":"3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129"}
Nov 02 20:15:33.150526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.150153    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3828ad418a03f3a488b4cb2d91ab6aeb3fefa6a933e7189dace4a5c60cb69129"
Nov 02 20:15:33.150526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.150444    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:33.151435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:33.151414    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:15:38.712431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.712386    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:15:38.713542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.713512    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" containerName="revoke-keys"
Nov 02 20:15:38.713542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.713537    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" containerName="revoke-keys"
Nov 02 20:15:38.722454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.721905    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" containerName="revoke-keys"
Nov 02 20:15:38.726547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.726521    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:38.779289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.779241    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:15:38.810959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.810707    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f75722a\""
Nov 02 20:15:38.810959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.810917    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7f75722a-dockercfg-wsjqs\""
Nov 02 20:15:38.873052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.872972    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:38.873484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.873449    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp97\" (UniqueName: \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:38.975343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.975246    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:38.990805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.978105    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp97\" (UniqueName: \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:39.074803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:38.976473    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:39.218652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:39.218598    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp97\" (UniqueName: \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\") pod \"payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:39.223604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:39.223579    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:39.700566 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:15:39.700528    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9136f10a_994a_4b13_bfb1_50bd5bee2ac1.slice/crio-b8daa51bcff16adf9404aad79c74e6107acdadcbb2b4416bdfa4939937046eba WatchSource:0}: Error finding container b8daa51bcff16adf9404aad79c74e6107acdadcbb2b4416bdfa4939937046eba: Status 404 returned error can't find the container with id b8daa51bcff16adf9404aad79c74e6107acdadcbb2b4416bdfa4939937046eba
Nov 02 20:15:39.702173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:39.702148    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:15:39.949904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:39.949823    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:15:40.277035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:40.276938    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" event={"ID":"9136f10a-994a-4b13-bfb1-50bd5bee2ac1","Type":"ContainerStarted","Data":"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"}
Nov 02 20:15:40.277035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:40.277007    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" event={"ID":"9136f10a-994a-4b13-bfb1-50bd5bee2ac1","Type":"ContainerStarted","Data":"b8daa51bcff16adf9404aad79c74e6107acdadcbb2b4416bdfa4939937046eba"}
Nov 02 20:15:40.277271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:40.277167    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:40.385163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:40.385105    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" podStartSLOduration=2.385089975 podStartE2EDuration="2.385089975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:40.383587409 +0000 UTC m=+1429.257858298" watchObservedRunningTime="2026-11-02 20:15:40.385089975 +0000 UTC m=+1429.259360862"
Nov 02 20:15:46.336812 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:15:46.320425    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:15:48.480211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.478961    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:48.525935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.525878    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:48.526671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.526638    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 20:15:48.557808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.529442    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:48.558187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.531491    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:48.559317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.531956    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:15:48.700961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:48.700893    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": context deadline exceeded"
Nov 02 20:15:49.258043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.252918    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:49.280831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.259770    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 02 20:15:49.479954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.479902    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:49.563027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.562926    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:15:49.613803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.586131    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:15:49.710749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.709191    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:15:49.710749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.617399    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:15:49.710749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.630325    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:15:49.710749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.630280    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:15:49.710749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.709708    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:15:49.734821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.731820    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:15:49.734821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:49.731897    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:15:50.170489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:50.170454    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:15:50.192641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:50.192410    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:15:50.282599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:50.282566    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:15:50.377920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:50.377738    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:15:50.417306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:50.417255    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:15:51.342014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.341972    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:15:51.784739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.784708    5099 scope.go:122] "RemoveContainer" containerID="deb6a0a2158d29d02f31cc65eb6a04898c9f69762a4d35c99d348cc5a22a8335"
Nov 02 20:15:51.796018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.795994    5099 scope.go:122] "RemoveContainer" containerID="8d5394459a4397687d28d77cbc70ea2358f5b5c021265fcb90a91b76fe4145e4"
Nov 02 20:15:51.805848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.805820    5099 scope.go:122] "RemoveContainer" containerID="9ad4e8492ee1f9f75bebd5998798572116d5e32ce71fd4422a8da86d54eaefec"
Nov 02 20:15:51.816246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.816221    5099 scope.go:122] "RemoveContainer" containerID="a867cf005e7118a0de6ba4386a888afe71bff4e71c6647f70dfe44b589f60e93"
Nov 02 20:15:51.826036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.826016    5099 scope.go:122] "RemoveContainer" containerID="dd563e9839d6e9c1c53b64c0a78d6df5067d82b33a047429787cac41f345ee2d"
Nov 02 20:15:51.835549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.835532    5099 scope.go:122] "RemoveContainer" containerID="01af96e1a291b1561b1924bfdc54960d998f5f5bd5e63e19381e362b32a799ee"
Nov 02 20:15:51.844946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.844924    5099 scope.go:122] "RemoveContainer" containerID="3c2f20c637ea93fe50308be08e648f7a256c976dc077e437eefbb9e3aa650fcb"
Nov 02 20:15:51.858232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.858212    5099 scope.go:122] "RemoveContainer" containerID="efa1e71987533e29150d38f615e66aba1eec505a44fff2e8ee8d7309a49d751d"
Nov 02 20:15:51.868570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.868545    5099 scope.go:122] "RemoveContainer" containerID="e06d4df4dfa14cef9f72b0797ec2243a7f36b1cd7df76d6e97959e7185a57910"
Nov 02 20:15:51.879435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:15:51.879414    5099 scope.go:122] "RemoveContainer" containerID="9ff820f2482b2be2e2c79cfb466479b8bccac916cde7ad811205da5c53cf34a8"
Nov 02 20:16:04.650098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.650058    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:16:04.657164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.657139    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677291    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677425    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677510    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677564    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677641    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677726    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677796 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677769    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cx88\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677822    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.677873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.677843    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.706546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.706515    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\""
Nov 02 20:16:04.759856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.759819    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:16:04.779908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.779873    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.779972    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780035    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780101    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780155    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780282    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780325    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cx88\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780362    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780392    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780405    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780832    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780873    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.780953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.780916    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.781184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.781123    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.784357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.783695    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.784975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.784941    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.835294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.835261    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.835442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.835325    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cx88\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\") pod \"maas-default-gateway-openshift-default-696f47c6c-htsgc\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:04.974955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:04.974870    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:05.404649 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:16:05.404605    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af1733_2694_4a58_9a8c_3758912188bd.slice/crio-ca286224b47926a23341dac06739c6aa3d4e2ec32d9959c8d75a8293414557d6 WatchSource:0}: Error finding container ca286224b47926a23341dac06739c6aa3d4e2ec32d9959c8d75a8293414557d6: Status 404 returned error can't find the container with id ca286224b47926a23341dac06739c6aa3d4e2ec32d9959c8d75a8293414557d6
Nov 02 20:16:05.406271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.406246    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:16:05.407371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.407342    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:05.407431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.407420    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:05.407471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.407449    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:05.532265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.532172    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" event={"ID":"24af1733-2694-4a58-9a8c-3758912188bd","Type":"ContainerStarted","Data":"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"}
Nov 02 20:16:05.532602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.532575    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" event={"ID":"24af1733-2694-4a58-9a8c-3758912188bd","Type":"ContainerStarted","Data":"ca286224b47926a23341dac06739c6aa3d4e2ec32d9959c8d75a8293414557d6"}
Nov 02 20:16:05.650927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.650801    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podStartSLOduration=1.650765378 podStartE2EDuration="1.650765378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:05.644361975 +0000 UTC m=+1454.518632864" watchObservedRunningTime="2026-11-02 20:16:05.650765378 +0000 UTC m=+1454.525036270"
Nov 02 20:16:05.975112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:05.975018    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:06.977599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:06.977557    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:06.978075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:06.977632    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:07.975862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:07.975819    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:07.976144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:07.975885    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:08.976291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:08.976247    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:08.976846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:08.976317    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:09.975893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:09.975861    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:09.976125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:09.975914    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:10.975491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:10.975448    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:10.975961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:10.975918    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:11.975400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:11.975352    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:11.975583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:11.975440    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:12.975262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:12.975222    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:12.975479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:12.975302    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:13.976344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:13.976293    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 20:16:13.976882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:13.976365    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 20:16:15.687279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:15.687240    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:15.693221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:15.693193    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:15.698017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:15.697983    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:16:16.592174 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:16:16.592117    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:16:26.106284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.106245    5099 generic.go:356] "Generic (PLEG): container finished" podID="22dbabb9-86de-41ad-887e-735a9f33d377" containerID="d6e07fc020ab671ae0dc955862d82c3d3292f70f4768bace06581231276ae371" exitCode=0
Nov 02 20:16:26.106864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.106546    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerDied","Data":"d6e07fc020ab671ae0dc955862d82c3d3292f70f4768bace06581231276ae371"}
Nov 02 20:16:26.107116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.107095    5099 scope.go:122] "RemoveContainer" containerID="d6e07fc020ab671ae0dc955862d82c3d3292f70f4768bace06581231276ae371"
Nov 02 20:16:26.108866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.108834    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:26.108959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.108904    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:26.108959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:26.108936    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:27.117235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:27.117197    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerStarted","Data":"2e5d2bcb2a15251cdf102232b430053368212f6001d89e1bf2158db9f0b492eb"}
Nov 02 20:16:27.955190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:27.955157    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:16:27.955922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:27.955890    5099 patch_prober.go:29] interesting pod/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:16:27.956131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:27.956103    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 20:16:28.948757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:28.948273    5099 patch_prober.go:29] interesting pod/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:16:28.948757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:28.948342    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 20:16:29.950118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:29.949916    5099 patch_prober.go:29] interesting pod/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:16:29.950698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:29.950665    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 20:16:30.962302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:30.962261    5099 patch_prober.go:29] interesting pod/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:16:30.962899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:30.962326    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 20:16:31.947238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:31.947197    5099 patch_prober.go:29] interesting pod/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 20:16:31.947549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:31.947517    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 20:16:32.951667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:32.951635    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:16:32.952255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:32.952014    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:16:32.952696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:32.952675    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:16:34.960388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:34.960349    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:16:34.966022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:34.965998    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.047969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.047935    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"]
Nov 02 20:16:35.062521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.062480    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.062691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.062527    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.062691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.062644    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.062691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.062661    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.062691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.062687    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.063035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.063004    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.063154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.063072    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.063154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.063145    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.063267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.063195    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5rg\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.069187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.069161    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ca7551b-hc78l"]
Nov 02 20:16:35.075325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.075299    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:16:35.269828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.269311    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.271961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.271932    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.274418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.274393    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.274643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.274624    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.275034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.275001    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.275946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.275922    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.276227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.276207    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.276384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.276362    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5rg\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.276681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.276606    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.309687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.305144    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.309687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.305794    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.309687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.306062    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.309687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.307238    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.309687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.276909    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.329253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.329216    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2debddb-fc95-4f28-b340-22c2aba1de2c" path="/var/lib/kubelet/pods/d2debddb-fc95-4f28-b340-22c2aba1de2c/volumes"
Nov 02 20:16:35.340480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.340428    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.342135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.342107    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.360501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.360468    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.361158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.361134    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5rg\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\") pod \"maas-default-gateway-openshift-default-696f47c6c-mkzm2\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:35.581014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:35.580927    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:36.021857 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:16:36.021819    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf64d0f7_370d_4eca_9858_39a41186c0a1.slice/crio-681a06d3d420b907ccc438d66f1b643ee199babc16188942318288733ec2ff8e WatchSource:0}: Error finding container 681a06d3d420b907ccc438d66f1b643ee199babc16188942318288733ec2ff8e: Status 404 returned error can't find the container with id 681a06d3d420b907ccc438d66f1b643ee199babc16188942318288733ec2ff8e
Nov 02 20:16:36.023422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.023395    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:16:36.025087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.025042    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:36.025177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.025134    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:36.025226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.025180    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:16:36.356101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.356003    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" event={"ID":"bf64d0f7-370d-4eca-9858-39a41186c0a1","Type":"ContainerStarted","Data":"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"}
Nov 02 20:16:36.356101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.356061    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" event={"ID":"bf64d0f7-370d-4eca-9858-39a41186c0a1","Type":"ContainerStarted","Data":"681a06d3d420b907ccc438d66f1b643ee199babc16188942318288733ec2ff8e"}
Nov 02 20:16:36.468983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.468929    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podStartSLOduration=2.468913981 podStartE2EDuration="2.468913981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:36.465326635 +0000 UTC m=+1485.339597524" watchObservedRunningTime="2026-11-02 20:16:36.468913981 +0000 UTC m=+1485.343184913"
Nov 02 20:16:36.581609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:36.581565    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:37.583226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:37.583189    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:37.583625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:37.583246    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:38.581485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:38.581448    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:38.581678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:38.581505    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:39.581579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:39.581542    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:39.582053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:39.581617    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:40.581965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:40.581931    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:40.582416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:40.582000    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:41.581413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:41.581376    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:41.581622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:41.581451    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:42.582137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:42.582094    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:42.582550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:42.582169    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:43.581606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:43.581562    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 20:16:43.581976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:43.581935    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 20:16:44.762093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:44.762057    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:44.763397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:44.763369    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:44.764430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:44.764405    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:16:46.096179 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:16:46.096140    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:16:49.046580 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:49.046540    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"]
Nov 02 20:16:49.093611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:49.093576    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fe5680-jm2hx"]
Nov 02 20:16:49.262911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:49.262878    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f766afad-1b95-4cee-937b-e2352f7e930b" path="/var/lib/kubelet/pods/f766afad-1b95-4cee-937b-e2352f7e930b/volumes"
Nov 02 20:16:51.685033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:51.684995    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:16:51.704425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:51.704401    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:16:54.100027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.098882    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"]
Nov 02 20:16:54.107198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.107169    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.157414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.157367    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"]
Nov 02 20:16:54.195436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.195265    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:16:54.195436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.195386    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:16:54.260285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.260248    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsgd\" (UniqueName: \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.260456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.260376    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.260503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.260478    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.361919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.361890    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.362106 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.362088    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsgd\" (UniqueName: \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.362240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.362213    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.362601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.362577    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.364480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.364461    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.416057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.416026    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsgd\" (UniqueName: \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\") pod \"maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.448579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.448539    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"]
Nov 02 20:16:54.453800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.453763    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.462941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.462819    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:54.506853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.506816    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"]
Nov 02 20:16:54.565983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.565771    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8n5q\" (UniqueName: \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.565983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.565986    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.566364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.566055    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.672857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.671542    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8n5q\" (UniqueName: \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.672857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.671743    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.672857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.671848    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.674060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.673266    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.676902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.676842    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.727435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.727398    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8n5q\" (UniqueName: \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\") pod \"maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.771233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.771193    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:54.871117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:54.853966    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"]
Nov 02 20:16:55.203444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:55.203409    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"]
Nov 02 20:16:55.280816 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:16:55.277920    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a141908_bcae_45ca_97a6_071a6ea3dd98.slice/crio-299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14 WatchSource:0}: Error finding container 299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14: Status 404 returned error can't find the container with id 299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14
Nov 02 20:16:55.698658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:55.698612    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c" event={"ID":"2a141908-bcae-45ca-97a6-071a6ea3dd98","Type":"ContainerStarted","Data":"299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14"}
Nov 02 20:16:55.700402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:55.700375    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2" event={"ID":"bec78d69-10b5-49eb-8fad-8182312b8047","Type":"ContainerStarted","Data":"0a1b29ec036fc9848247853698c77853c15330813de2b65465497ee26206b680"}
Nov 02 20:16:55.700468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:55.700422    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2" event={"ID":"bec78d69-10b5-49eb-8fad-8182312b8047","Type":"ContainerStarted","Data":"0a4df449a275b4c64e76d9da403e4be9e9a36b544acdf74aa97374b6b2abe8a5"}
Nov 02 20:16:55.889802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:55.884288    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2" podStartSLOduration=1.8842642870000001 podStartE2EDuration="1.884264287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:55.865478719 +0000 UTC m=+1504.739749608" watchObservedRunningTime="2026-11-02 20:16:55.884264287 +0000 UTC m=+1504.758535156"
Nov 02 20:16:56.713663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:56.713617    5099 generic.go:356] "Generic (PLEG): container finished" podID="2a141908-bcae-45ca-97a6-071a6ea3dd98" containerID="62ec56e1ccc9c1ccf80173c1e16f510659f7cbd2c8c663437eda46da407d61c9" exitCode=0
Nov 02 20:16:56.714062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:56.713697    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c" event={"ID":"2a141908-bcae-45ca-97a6-071a6ea3dd98","Type":"ContainerDied","Data":"62ec56e1ccc9c1ccf80173c1e16f510659f7cbd2c8c663437eda46da407d61c9"}
Nov 02 20:16:56.715578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:56.715552    5099 generic.go:356] "Generic (PLEG): container finished" podID="bec78d69-10b5-49eb-8fad-8182312b8047" containerID="0a1b29ec036fc9848247853698c77853c15330813de2b65465497ee26206b680" exitCode=0
Nov 02 20:16:56.715701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:56.715612    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2" event={"ID":"bec78d69-10b5-49eb-8fad-8182312b8047","Type":"ContainerDied","Data":"0a1b29ec036fc9848247853698c77853c15330813de2b65465497ee26206b680"}
Nov 02 20:16:58.109521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.109488    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:58.191806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.191438    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\") pod \"2a141908-bcae-45ca-97a6-071a6ea3dd98\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") "
Nov 02 20:16:58.191806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.191634    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\" (UniqueName: \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\") pod \"2a141908-bcae-45ca-97a6-071a6ea3dd98\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") "
Nov 02 20:16:58.191806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.191774    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\") pod \"2a141908-bcae-45ca-97a6-071a6ea3dd98\" (UID: \"2a141908-bcae-45ca-97a6-071a6ea3dd98\") "
Nov 02 20:16:58.192584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.192557    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca" pod "2a141908-bcae-45ca-97a6-071a6ea3dd98" (UID: "2a141908-bcae-45ca-97a6-071a6ea3dd98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:58.195003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.194974    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q" pod "2a141908-bcae-45ca-97a6-071a6ea3dd98" (UID: "2a141908-bcae-45ca-97a6-071a6ea3dd98"). InnerVolumeSpecName "kube-api-access-v8n5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:58.195585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.195547    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca" pod "2a141908-bcae-45ca-97a6-071a6ea3dd98" (UID: "2a141908-bcae-45ca-97a6-071a6ea3dd98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:58.220891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.220864    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:58.296122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.296028    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\" (UniqueName: \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\") pod \"bec78d69-10b5-49eb-8fad-8182312b8047\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") "
Nov 02 20:16:58.296306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.296242    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\") pod \"bec78d69-10b5-49eb-8fad-8182312b8047\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") "
Nov 02 20:16:58.296306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.296287    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\") pod \"bec78d69-10b5-49eb-8fad-8182312b8047\" (UID: \"bec78d69-10b5-49eb-8fad-8182312b8047\") "
Nov 02 20:16:58.297232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.297206    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.297356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.297236    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8n5q\" (UniqueName: \"kubernetes.io/projected/2a141908-bcae-45ca-97a6-071a6ea3dd98-kube-api-access-v8n5q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.297356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.297253    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a141908-bcae-45ca-97a6-071a6ea3dd98-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.297631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.297609    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca" pod "bec78d69-10b5-49eb-8fad-8182312b8047" (UID: "bec78d69-10b5-49eb-8fad-8182312b8047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:16:58.297946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.297922    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd" pod "bec78d69-10b5-49eb-8fad-8182312b8047" (UID: "bec78d69-10b5-49eb-8fad-8182312b8047"). InnerVolumeSpecName "kube-api-access-bfsgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:16:58.299715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.299688    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca" pod "bec78d69-10b5-49eb-8fad-8182312b8047" (UID: "bec78d69-10b5-49eb-8fad-8182312b8047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:16:58.504811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.463316    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsgd\" (UniqueName: \"kubernetes.io/projected/bec78d69-10b5-49eb-8fad-8182312b8047-kube-api-access-bfsgd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.505111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.505080    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.505253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.505235    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec78d69-10b5-49eb-8fad-8182312b8047-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:16:58.734256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.734231    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"
Nov 02 20:16:58.734533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.734228    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c" event={"ID":"2a141908-bcae-45ca-97a6-071a6ea3dd98","Type":"ContainerDied","Data":"299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14"}
Nov 02 20:16:58.734625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.734566    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="299b003c211362999afb352ebcc7cba547044e15ce02d25ddea1462d24effb14"
Nov 02 20:16:58.735970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.735948    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"
Nov 02 20:16:58.736086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.735970    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2" event={"ID":"bec78d69-10b5-49eb-8fad-8182312b8047","Type":"ContainerDied","Data":"0a4df449a275b4c64e76d9da403e4be9e9a36b544acdf74aa97374b6b2abe8a5"}
Nov 02 20:16:58.736086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:16:58.736006    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a4df449a275b4c64e76d9da403e4be9e9a36b544acdf74aa97374b6b2abe8a5"
Nov 02 20:17:04.191949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.191910    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:17:04.192455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.191974    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:17:04.832524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.832434    5099 generic.go:356] "Generic (PLEG): container finished" podID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerID="8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40" exitCode=0
Nov 02 20:17:04.832524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.832487    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerDied","Data":"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"}
Nov 02 20:17:04.833089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.833071    5099 scope.go:122] "RemoveContainer" containerID="8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"
Nov 02 20:17:04.834971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.834944    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:04.835063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.835014    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:04.835063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:04.835044    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.880733    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerStarted","Data":"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"}
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.882990    5099 generic.go:356] "Generic (PLEG): container finished" podID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerID="1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7" exitCode=0
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.883315    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerDied","Data":"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"}
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.883869    5099 scope.go:122] "RemoveContainer" containerID="1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.886081    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.886172    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:05.887807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:05.886243    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:06.098802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:06.096989    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:06.098802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:06.097051    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:06.098802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:06.097127    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:17:06.898869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:06.898830    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerStarted","Data":"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"}
Nov 02 20:17:07.068846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:07.068471    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:07.068846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:07.068561    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:07.822235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:07.822185    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:17:07.839962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:07.823717    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:07.840305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:07.840276    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:08.079422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.079338    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:08.080053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.080009    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:08.294807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.290562    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:17:08.715268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.715228    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:17:08.821302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.821252    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:08.821526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.821335    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:08.879807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:08.875885    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:17:09.070298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:09.070207    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:09.070558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:09.070526    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:09.820842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:09.820811    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:09.821321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:09.821278    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:10.079863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.078760    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:10.079863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.078853    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:10.525806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.521614    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"]
Nov 02 20:17:10.525806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.522874    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2a141908-bcae-45ca-97a6-071a6ea3dd98" containerName="revoke-keys"
Nov 02 20:17:10.525806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.522896    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a141908-bcae-45ca-97a6-071a6ea3dd98" containerName="revoke-keys"
Nov 02 20:17:10.525806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.522962    5099 cpu_manager.go:414] "containerMap: removing container" podUID="bec78d69-10b5-49eb-8fad-8182312b8047" containerName="revoke-keys"
Nov 02 20:17:10.525806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.522970    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec78d69-10b5-49eb-8fad-8182312b8047" containerName="revoke-keys"
Nov 02 20:17:10.538606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.538569    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec78d69-10b5-49eb-8fad-8182312b8047" containerName="revoke-keys"
Nov 02 20:17:10.538890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.538871    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a141908-bcae-45ca-97a6-071a6ea3dd98" containerName="revoke-keys"
Nov 02 20:17:10.659930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.659899    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:10.677569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.677504    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"]
Nov 02 20:17:10.886524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.871436    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2b8z\" (UniqueName: \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:10.910820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.887479    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:10.911813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.911756    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:10.926635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.926598    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:10.929086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.929003    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:10.994808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.969737    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:17:10.995156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:10.995124    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:17:11.049363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.020892    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2b8z\" (UniqueName: \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.053774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.053696    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.063138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.063100    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.070806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.064228    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.222856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.187196    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:11.222856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.187293    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:11.535214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.469909    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.535214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.471015    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2b8z\" (UniqueName: \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\") pod \"maas-api-revoke-sub-83492e8e-rcc94\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.535214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.485472    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:17:11.607116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.607076    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:11.825978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.825892    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:11.826229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:11.826201    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:12.113103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:12.112903    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:12.113103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:12.112994    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:12.797644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:12.797518    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"]
Nov 02 20:17:12.798819 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:17:12.798771    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee603e47_f86a_4fc9_b654_78ee4f3dfbd0.slice/crio-339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4 WatchSource:0}: Error finding container 339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4: Status 404 returned error can't find the container with id 339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4
Nov 02 20:17:12.820776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:12.820749    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:12.820893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:12.820813    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:13.071389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.071295    5099 patch_prober.go:29] interesting pod/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 20:17:13.071648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.071621    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 20:17:13.090418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.090360    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"]
Nov 02 20:17:13.100769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.100123    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda036ed-ct6xx"]
Nov 02 20:17:13.288736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.288706    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4abe0f-01c2-45fd-934f-b4c46d60a0c6" path="/var/lib/kubelet/pods/8f4abe0f-01c2-45fd-934f-b4c46d60a0c6/volumes"
Nov 02 20:17:13.432877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.432836    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94" event={"ID":"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0","Type":"ContainerStarted","Data":"8b120e642c00c4444d924069dd8e6546fe016467163cf3d659af9e9d554847ea"}
Nov 02 20:17:13.433058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.432906    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94" event={"ID":"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0","Type":"ContainerStarted","Data":"339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4"}
Nov 02 20:17:13.572716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.572649    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94" podStartSLOduration=3.5726311170000002 podStartE2EDuration="3.572631117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:13.543938978 +0000 UTC m=+1522.418209866" watchObservedRunningTime="2026-11-02 20:17:13.572631117 +0000 UTC m=+1522.446902002"
Nov 02 20:17:13.727252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.727139    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" podUID="44432bb6-6c06-4654-85a2-fbc34db18034" containerName="payload-pre-processing" containerID="cri-o://7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80" gracePeriod=25
Nov 02 20:17:13.821093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.821063    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:13.821255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:13.821115    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:14.072223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.072193    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:17:14.072759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.072734    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:17:14.073710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.073692    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:17:14.118167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.118143    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:17:14.263878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.263113    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\") pod \"44432bb6-6c06-4654-85a2-fbc34db18034\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") "
Nov 02 20:17:14.263878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.263236    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\" (UniqueName: \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\") pod \"44432bb6-6c06-4654-85a2-fbc34db18034\" (UID: \"44432bb6-6c06-4654-85a2-fbc34db18034\") "
Nov 02 20:17:14.263878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.263670    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume" pod "44432bb6-6c06-4654-85a2-fbc34db18034" (UID: "44432bb6-6c06-4654-85a2-fbc34db18034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:14.263878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.263861    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44432bb6-6c06-4654-85a2-fbc34db18034-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.265693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.265671    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475" pod "44432bb6-6c06-4654-85a2-fbc34db18034" (UID: "44432bb6-6c06-4654-85a2-fbc34db18034"). InnerVolumeSpecName "kube-api-access-55475". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:14.370090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.370000    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55475\" (UniqueName: \"kubernetes.io/projected/44432bb6-6c06-4654-85a2-fbc34db18034-kube-api-access-55475\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:14.487771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.487728    5099 generic.go:356] "Generic (PLEG): container finished" podID="44432bb6-6c06-4654-85a2-fbc34db18034" containerID="7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80" exitCode=0
Nov 02 20:17:14.488099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.488070    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" event={"ID":"44432bb6-6c06-4654-85a2-fbc34db18034","Type":"ContainerDied","Data":"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"}
Nov 02 20:17:14.488199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.488139    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj" event={"ID":"44432bb6-6c06-4654-85a2-fbc34db18034","Type":"ContainerDied","Data":"e9f6b505f2dd3bb407afa96e8e8863c528d7ca77d5085a1a4c876cb93259fac0"}
Nov 02 20:17:14.488199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.488171    5099 scope.go:122] "RemoveContainer" containerID="7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"
Nov 02 20:17:14.488359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.488341    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"
Nov 02 20:17:14.497862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.497830    5099 generic.go:356] "Generic (PLEG): container finished" podID="ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" containerID="8b120e642c00c4444d924069dd8e6546fe016467163cf3d659af9e9d554847ea" exitCode=0
Nov 02 20:17:14.499180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.498280    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94" event={"ID":"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0","Type":"ContainerDied","Data":"8b120e642c00c4444d924069dd8e6546fe016467163cf3d659af9e9d554847ea"}
Nov 02 20:17:14.512822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.512797    5099 scope.go:122] "RemoveContainer" containerID="7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"
Nov 02 20:17:14.513222 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:17:14.513195    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80\": container with ID starting with 7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80 not found: ID does not exist" containerID="7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"
Nov 02 20:17:14.513310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.513236    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80"} err="failed to get container status \"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80\": rpc error: code = NotFound desc = could not find container \"7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80\": container with ID starting with 7dff9b29e617891ccf8e06abd61c331c881354644d0875067beb8faca941bd80 not found: ID does not exist"
Nov 02 20:17:14.769462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.769421    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:17:14.774612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.774584    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2b4e70c-6c6cbb66b7-vc4wj"]
Nov 02 20:17:14.820814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.820759    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 02 20:17:14.820957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:14.820846    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 02 20:17:15.261118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.261084    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44432bb6-6c06-4654-85a2-fbc34db18034" path="/var/lib/kubelet/pods/44432bb6-6c06-4654-85a2-fbc34db18034/volumes"
Nov 02 20:17:15.705538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.705512    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:15.800349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.800316    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\") pod \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") "
Nov 02 20:17:15.800514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.800426    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\") pod \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") "
Nov 02 20:17:15.800621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.800607    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\" (UniqueName: \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\") pod \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\" (UID: \"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0\") "
Nov 02 20:17:15.800856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.800823    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca" pod "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" (UID: "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:15.801253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.801231    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.802570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.802547    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca" pod "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" (UID: "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:15.802735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.802714    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z" pod "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" (UID: "ee603e47-f86a-4fc9-b654-78ee4f3dfbd0"). InnerVolumeSpecName "kube-api-access-c2b8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:15.826370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.826294    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:17:15.827838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.827768    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:17:15.902773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.902736    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2b8z\" (UniqueName: \"kubernetes.io/projected/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-kube-api-access-c2b8z\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:15.902773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:15.902764    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:16.108852 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:17:16.108751    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:17:16.238594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.238562    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"]
Nov 02 20:17:16.239431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239408    5099 cpu_manager.go:414] "containerMap: removing container" podUID="44432bb6-6c06-4654-85a2-fbc34db18034" containerName="payload-pre-processing"
Nov 02 20:17:16.239564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239432    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44432bb6-6c06-4654-85a2-fbc34db18034" containerName="payload-pre-processing"
Nov 02 20:17:16.239564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239510    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" containerName="revoke-keys"
Nov 02 20:17:16.239564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239521    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" containerName="revoke-keys"
Nov 02 20:17:16.239725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239693    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" containerName="revoke-keys"
Nov 02 20:17:16.239725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.239714    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="44432bb6-6c06-4654-85a2-fbc34db18034" containerName="payload-pre-processing"
Nov 02 20:17:16.243748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.243728    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.295638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.295603    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"]
Nov 02 20:17:16.410171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.410132    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.410385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.410211    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.410544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.410525    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqndh\" (UniqueName: \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.511704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.511668    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.511885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.511749    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.511957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.511885    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqndh\" (UniqueName: \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.512510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.512491    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.514546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.514518    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.519840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.519811    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94" event={"ID":"ee603e47-f86a-4fc9-b654-78ee4f3dfbd0","Type":"ContainerDied","Data":"339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4"}
Nov 02 20:17:16.519945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.519861    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339c6a5dab47fcfad2d21ad5b27938334f2c86dc57eaa17f25a85b2e31470da4"
Nov 02 20:17:16.519945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.519826    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"
Nov 02 20:17:16.569677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.567325    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqndh\" (UniqueName: \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\") pod \"maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:16.862510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:16.862476    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:17.359815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:17.359770    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"]
Nov 02 20:17:17.531711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:17.531674    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd" event={"ID":"0f71ca99-c569-41a0-8007-9437d191f6aa","Type":"ContainerStarted","Data":"4ebb21724e90448fe80c9429895079fd700204074ec56f1a41a0d09e4c668d6c"}
Nov 02 20:17:18.542461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:18.542377    5099 generic.go:356] "Generic (PLEG): container finished" podID="0f71ca99-c569-41a0-8007-9437d191f6aa" containerID="d1ed4d83f31d065ea02b997b10a634627f158825d3dd4157c6368ed732d087f1" exitCode=0
Nov 02 20:17:18.542876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:18.542467    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd" event={"ID":"0f71ca99-c569-41a0-8007-9437d191f6aa","Type":"ContainerDied","Data":"d1ed4d83f31d065ea02b997b10a634627f158825d3dd4157c6368ed732d087f1"}
Nov 02 20:17:19.727258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.727229    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:19.844049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.843967    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\") pod \"0f71ca99-c569-41a0-8007-9437d191f6aa\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") "
Nov 02 20:17:19.844200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.844107    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\" (UniqueName: \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\") pod \"0f71ca99-c569-41a0-8007-9437d191f6aa\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") "
Nov 02 20:17:19.844273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.844256    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\") pod \"0f71ca99-c569-41a0-8007-9437d191f6aa\" (UID: \"0f71ca99-c569-41a0-8007-9437d191f6aa\") "
Nov 02 20:17:19.844417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.844390    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca" pod "0f71ca99-c569-41a0-8007-9437d191f6aa" (UID: "0f71ca99-c569-41a0-8007-9437d191f6aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:19.844980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.844955    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:19.846980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.846948    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh" pod "0f71ca99-c569-41a0-8007-9437d191f6aa" (UID: "0f71ca99-c569-41a0-8007-9437d191f6aa"). InnerVolumeSpecName "kube-api-access-dqndh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:19.848940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.848911    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca" pod "0f71ca99-c569-41a0-8007-9437d191f6aa" (UID: "0f71ca99-c569-41a0-8007-9437d191f6aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:17:19.946836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.946767    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqndh\" (UniqueName: \"kubernetes.io/projected/0f71ca99-c569-41a0-8007-9437d191f6aa-kube-api-access-dqndh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:19.946836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:19.946835    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f71ca99-c569-41a0-8007-9437d191f6aa-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:20.572551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:20.572510    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"
Nov 02 20:17:20.572551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:20.572535    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd" event={"ID":"0f71ca99-c569-41a0-8007-9437d191f6aa","Type":"ContainerDied","Data":"4ebb21724e90448fe80c9429895079fd700204074ec56f1a41a0d09e4c668d6c"}
Nov 02 20:17:20.572891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:20.572586    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ebb21724e90448fe80c9429895079fd700204074ec56f1a41a0d09e4c668d6c"
Nov 02 20:17:28.011757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:28.011719    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:17:29.049686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:29.049639    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"]
Nov 02 20:17:29.054950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:29.054924    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddccc649-pkmpz"]
Nov 02 20:17:29.260061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:29.260029    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27544f53-d2fa-4f15-94dc-2893028ae92a" path="/var/lib/kubelet/pods/27544f53-d2fa-4f15-94dc-2893028ae92a/volumes"
Nov 02 20:17:33.012519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.012448    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" podUID="77819e2b-a43a-45d0-b128-81133cd8b063" containerName="payload-pre-processing" containerID="cri-o://93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3" gracePeriod=25
Nov 02 20:17:33.338990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.338949    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:17:33.441894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.441858    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\") pod \"77819e2b-a43a-45d0-b128-81133cd8b063\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") "
Nov 02 20:17:33.442267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.442242    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume" pod "77819e2b-a43a-45d0-b128-81133cd8b063" (UID: "77819e2b-a43a-45d0-b128-81133cd8b063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:33.442393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.442244    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\" (UniqueName: \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\") pod \"77819e2b-a43a-45d0-b128-81133cd8b063\" (UID: \"77819e2b-a43a-45d0-b128-81133cd8b063\") "
Nov 02 20:17:33.442954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.442928    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819e2b-a43a-45d0-b128-81133cd8b063-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:33.444446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.444422    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4" pod "77819e2b-a43a-45d0-b128-81133cd8b063" (UID: "77819e2b-a43a-45d0-b128-81133cd8b063"). InnerVolumeSpecName "kube-api-access-7w6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:33.544098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.544063    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6f4\" (UniqueName: \"kubernetes.io/projected/77819e2b-a43a-45d0-b128-81133cd8b063-kube-api-access-7w6f4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:33.816079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.816043    5099 generic.go:356] "Generic (PLEG): container finished" podID="77819e2b-a43a-45d0-b128-81133cd8b063" containerID="93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3" exitCode=0
Nov 02 20:17:33.816265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.816088    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"
Nov 02 20:17:33.816265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.816119    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" event={"ID":"77819e2b-a43a-45d0-b128-81133cd8b063","Type":"ContainerDied","Data":"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"}
Nov 02 20:17:33.816265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.816171    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f" event={"ID":"77819e2b-a43a-45d0-b128-81133cd8b063","Type":"ContainerDied","Data":"e70afc0f34712f47163a0e390d3b522623b9c78b80a1393c1b31f7242a8dcc19"}
Nov 02 20:17:33.816265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.816194    5099 scope.go:122] "RemoveContainer" containerID="93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"
Nov 02 20:17:33.828542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.828518    5099 scope.go:122] "RemoveContainer" containerID="93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"
Nov 02 20:17:33.828834 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:17:33.828807    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3\": container with ID starting with 93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3 not found: ID does not exist" containerID="93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"
Nov 02 20:17:33.828891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.828841    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3"} err="failed to get container status \"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3\": rpc error: code = NotFound desc = could not find container \"93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3\": container with ID starting with 93b34d664703883e0d0b88a07741a8b068fdf3b6180ae5f54d5613d776551dc3 not found: ID does not exist"
Nov 02 20:17:33.975301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:33.975262    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:17:34.021950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:34.021915    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2d73d56c-5c69f8db47-9249f"]
Nov 02 20:17:35.260380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:35.260344    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77819e2b-a43a-45d0-b128-81133cd8b063" path="/var/lib/kubelet/pods/77819e2b-a43a-45d0-b128-81133cd8b063/volumes"
Nov 02 20:17:42.054536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:42.054480    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"]
Nov 02 20:17:42.128994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:42.128936    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2b4e70c-lt7w2"]
Nov 02 20:17:42.137959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:42.137930    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:17:42.138390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:42.138359    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy" containerID="cri-o://2e5d2bcb2a15251cdf102232b430053368212f6001d89e1bf2158db9f0b492eb" gracePeriod=30
Nov 02 20:17:43.218523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:43.213885    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"]
Nov 02 20:17:43.233961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:43.232471    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2ac9784d-tk94c"]
Nov 02 20:17:43.303148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:43.302239    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a141908-bcae-45ca-97a6-071a6ea3dd98" path="/var/lib/kubelet/pods/2a141908-bcae-45ca-97a6-071a6ea3dd98/volumes"
Nov 02 20:17:43.303148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:43.302743    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec78d69-10b5-49eb-8fad-8182312b8047" path="/var/lib/kubelet/pods/bec78d69-10b5-49eb-8fad-8182312b8047/volumes"
Nov 02 20:17:46.103586 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:17:46.103547    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:17:47.397417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.397382    5099 generic.go:356] "Generic (PLEG): container finished" podID="22dbabb9-86de-41ad-887e-735a9f33d377" containerID="2e5d2bcb2a15251cdf102232b430053368212f6001d89e1bf2158db9f0b492eb" exitCode=0
Nov 02 20:17:47.397944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.397436    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerDied","Data":"2e5d2bcb2a15251cdf102232b430053368212f6001d89e1bf2158db9f0b492eb"}
Nov 02 20:17:47.397944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.397484    5099 scope.go:122] "RemoveContainer" containerID="d6e07fc020ab671ae0dc955862d82c3d3292f70f4768bace06581231276ae371"
Nov 02 20:17:47.456194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.456164    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:17:47.504595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504560    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504636    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504755    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504797    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504845    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504886    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.504980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.504922    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.505094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505012    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:47.505094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505050    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:47.505214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505134    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.505338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505213    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\") pod \"22dbabb9-86de-41ad-887e-735a9f33d377\" (UID: \"22dbabb9-86de-41ad-887e-735a9f33d377\") "
Nov 02 20:17:47.505401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505135    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:47.505462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505446    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:47.505849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.505820    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:17:47.506160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.506131    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.506160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.506162    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dbabb9-86de-41ad-887e-735a9f33d377-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.506345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.506179    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.506345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.506195    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.506345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.506209    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.507646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.507621    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:17:47.507767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.507656    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:47.507767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.507683    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "kube-api-access-pcgmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:17:47.507767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.507724    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo" pod "22dbabb9-86de-41ad-887e-735a9f33d377" (UID: "22dbabb9-86de-41ad-887e-735a9f33d377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:17:47.607984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.607947    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dbabb9-86de-41ad-887e-735a9f33d377-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.607984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.607979    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dbabb9-86de-41ad-887e-735a9f33d377-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.608188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.607994    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcgmw\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-kube-api-access-pcgmw\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:47.608188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:47.608004    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dbabb9-86de-41ad-887e-735a9f33d377-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:17:48.407064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:48.407034    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"
Nov 02 20:17:48.407518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:48.407067    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh" event={"ID":"22dbabb9-86de-41ad-887e-735a9f33d377","Type":"ContainerDied","Data":"6eb50a1afce84540812ee3afbd099902746c8afaff83f9f2f74292a4193ac331"}
Nov 02 20:17:48.407518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:48.407127    5099 scope.go:122] "RemoveContainer" containerID="2e5d2bcb2a15251cdf102232b430053368212f6001d89e1bf2158db9f0b492eb"
Nov 02 20:17:48.592813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:48.592761    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:17:48.640574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:48.640535    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a2b4e70c-openshift-default-55bbf9f8d5-7ghsh"]
Nov 02 20:17:49.262948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:49.262911    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" path="/var/lib/kubelet/pods/22dbabb9-86de-41ad-887e-735a9f33d377/volumes"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.753863    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755189    5099 cpu_manager.go:414] "containerMap: removing container" podUID="77819e2b-a43a-45d0-b128-81133cd8b063" containerName="payload-pre-processing"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755214    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77819e2b-a43a-45d0-b128-81133cd8b063" containerName="payload-pre-processing"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755260    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0f71ca99-c569-41a0-8007-9437d191f6aa" containerName="revoke-keys"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755270    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f71ca99-c569-41a0-8007-9437d191f6aa" containerName="revoke-keys"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755294    5099 cpu_manager.go:414] "containerMap: removing container" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755302    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755315    5099 cpu_manager.go:414] "containerMap: removing container" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755323    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755530    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="77819e2b-a43a-45d0-b128-81133cd8b063" containerName="payload-pre-processing"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755551    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f71ca99-c569-41a0-8007-9437d191f6aa" containerName="revoke-keys"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755563    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.755819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.755572    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbabb9-86de-41ad-887e-735a9f33d377" containerName="istio-proxy"
Nov 02 20:17:50.778804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.772420    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.778804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.777653    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:17:50.848695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.848613    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:17:50.954661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.954614    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.954942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.954919    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjbb\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955197    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955332    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955459    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955564    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955752    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:50.955928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:50.955912    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.051895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.051806    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.132803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.122259    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.176706    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-13c1d955\""
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.176907    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-13c1d955-dockercfg-8mvx9\""
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.208589    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.208712    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.208767    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjbb\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.208928    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.208965    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.209022    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.209060    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.209183    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.209221    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.219070    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.219384    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.253024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.229229    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.320540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.320459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.401807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.374518    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.447311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.447263    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.453462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.423503    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.453958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.413426    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.469932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.469898    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.501364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.501326    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.506887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.506858    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjbb\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\") pod \"maas-default-gateway-openshift-default-696f47c6c-gmgt4\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.516717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.516685    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:17:51.517038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.517023    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:17:51.526444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.525402    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:17:51.526444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.525492    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.576851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.576755    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.577101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.576931    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.577101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.577051    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6729d\" (UniqueName: \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.577186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.577119    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.625469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.625434    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-13c1d955\""
Nov 02 20:17:51.628372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.628342    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.630684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.630662    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\") pod \"payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") " pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.679773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.679728    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.679959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.679865    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6729d\" (UniqueName: \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.680552 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.680532    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.727424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.727385    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:51.733959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.733930    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6729d\" (UniqueName: \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\") pod \"payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:51.822667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.822638    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-13c1d955-dockercfg-8mvx9\""
Nov 02 20:17:51.824761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.824734    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:51.843080 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:51.842865    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:52.003647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.003617    5099 scope.go:122] "RemoveContainer" containerID="1288411b0f52896b2188380c2bb0370e33d643bde0a20dfd36cb88108d91f345"
Nov 02 20:17:52.015954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.015933    5099 scope.go:122] "RemoveContainer" containerID="d5a4166b5829df45ef91bced412f6b44ed77568741eeec9009ce8b2af7eb22d2"
Nov 02 20:17:52.154412 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:17:52.154382    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ee9d4e_6a47_4b42_87a1_0d6c0f26341b.slice/crio-14049544758fbcabb3ef323f0295e78ea2698313072f22231d4bee0207f77469 WatchSource:0}: Error finding container 14049544758fbcabb3ef323f0295e78ea2698313072f22231d4bee0207f77469: Status 404 returned error can't find the container with id 14049544758fbcabb3ef323f0295e78ea2698313072f22231d4bee0207f77469
Nov 02 20:17:52.156316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.156293    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:17:52.157175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.157145    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:52.157243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.157208    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:52.157243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.157238    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:52.257198 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:17:52.257168    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d929d3f_2f5b_48fe_b2e6_e843e05d73b4.slice/crio-d615654cc8a18659d5540ca27ecd2480628cf20f0dc08c9d499908726f5afbeb WatchSource:0}: Error finding container d615654cc8a18659d5540ca27ecd2480628cf20f0dc08c9d499908726f5afbeb: Status 404 returned error can't find the container with id d615654cc8a18659d5540ca27ecd2480628cf20f0dc08c9d499908726f5afbeb
Nov 02 20:17:52.258307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.258283    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:17:52.259398 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:17:52.259361    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdcd4d9_91f7_4303_bbc3_7bda3aaafd73.slice/crio-c5f29da348328ea6fbcc3d2c33a3b3d9d19a7093964b19e4316a126ad9173791 WatchSource:0}: Error finding container c5f29da348328ea6fbcc3d2c33a3b3d9d19a7093964b19e4316a126ad9173791: Status 404 returned error can't find the container with id c5f29da348328ea6fbcc3d2c33a3b3d9d19a7093964b19e4316a126ad9173791
Nov 02 20:17:52.265614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.265590    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:17:52.791174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.791135    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" event={"ID":"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73","Type":"ContainerStarted","Data":"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"}
Nov 02 20:17:52.791375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.791195    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:17:52.791375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.791213    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" event={"ID":"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73","Type":"ContainerStarted","Data":"c5f29da348328ea6fbcc3d2c33a3b3d9d19a7093964b19e4316a126ad9173791"}
Nov 02 20:17:52.792931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.792899    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" event={"ID":"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4","Type":"ContainerStarted","Data":"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"}
Nov 02 20:17:52.793064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.792961    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:17:52.793064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.792987    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" event={"ID":"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4","Type":"ContainerStarted","Data":"d615654cc8a18659d5540ca27ecd2480628cf20f0dc08c9d499908726f5afbeb"}
Nov 02 20:17:52.794567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.794540    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" event={"ID":"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b","Type":"ContainerStarted","Data":"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"}
Nov 02 20:17:52.794693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.794581    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" event={"ID":"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b","Type":"ContainerStarted","Data":"14049544758fbcabb3ef323f0295e78ea2698313072f22231d4bee0207f77469"}
Nov 02 20:17:52.918644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:52.918583    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" podStartSLOduration=2.918570359 podStartE2EDuration="2.918570359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:52.915145529 +0000 UTC m=+1561.789416418" watchObservedRunningTime="2026-11-02 20:17:52.918570359 +0000 UTC m=+1561.792841241"
Nov 02 20:17:53.180805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:53.163369    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podStartSLOduration=3.163344665 podStartE2EDuration="3.163344665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:53.073506195 +0000 UTC m=+1561.947777084" watchObservedRunningTime="2026-11-02 20:17:53.163344665 +0000 UTC m=+1562.037615555"
Nov 02 20:17:53.207804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:53.207470    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" podStartSLOduration=3.207448963 podStartE2EDuration="3.207448963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:53.18998334 +0000 UTC m=+1562.064254229" watchObservedRunningTime="2026-11-02 20:17:53.207448963 +0000 UTC m=+1562.081719865"
Nov 02 20:17:53.728509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:53.728466    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:17:53.730031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:53.729994    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:53.730161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:53.730054    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:54.072494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.072413    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"]
Nov 02 20:17:54.112019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.111988    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"]
Nov 02 20:17:54.116904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.116880    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f594a5-jfzqn"]
Nov 02 20:17:54.121009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.120988    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1861e62b-m6cj2"]
Nov 02 20:17:54.421985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.421721    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:17:54.427505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.427482    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.489233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.489201    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.568998    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgm4\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569117    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569146    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569304    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.589803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569345    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.590905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.590878    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-55e81986-openshift-default-dockercfg-sv7q6\""
Nov 02 20:17:54.615809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.569448    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.615809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.595115    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.615809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.595233    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.702958    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703034    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703076    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703178    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703335    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703462    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703524    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgm4\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703640    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.703861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.703664    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.704465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.704260    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.723270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.723227    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.724133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.724103    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.728852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.724576    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.729323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.729294    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.729488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.729463    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:54.735440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.735402    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:54.772812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.771886    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.796809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.792901    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.805813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.797674    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgm4\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:54.842836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:54.833388    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\") pod \"e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") " pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:55.052016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.049395    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:55.287299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.286976    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d72231b-3111-4dbb-968c-608cac4c2b3b" path="/var/lib/kubelet/pods/3d72231b-3111-4dbb-968c-608cac4c2b3b/volumes"
Nov 02 20:17:55.288225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.287724    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44997a9-d55b-4f4d-b810-022c3aa74935" path="/var/lib/kubelet/pods/d44997a9-d55b-4f4d-b810-022c3aa74935/volumes"
Nov 02 20:17:55.666695 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:17:55.666660    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2b1b60_82f8_4641_8919_5eec77dae2c9.slice/crio-e5987d30f1139999599bf39e1d09233afbe429abec5851e7c8c6e4960a9026ee WatchSource:0}: Error finding container e5987d30f1139999599bf39e1d09233afbe429abec5851e7c8c6e4960a9026ee: Status 404 returned error can't find the container with id e5987d30f1139999599bf39e1d09233afbe429abec5851e7c8c6e4960a9026ee
Nov 02 20:17:55.667363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.667333    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:17:55.669409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.669372    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:55.669487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.669442    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:55.669487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.669469    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:17:55.728091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.728066    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:55.728235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.728113    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:55.877748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.877701    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerStarted","Data":"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"}
Nov 02 20:17:55.877938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.877820    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerStarted","Data":"e5987d30f1139999599bf39e1d09233afbe429abec5851e7c8c6e4960a9026ee"}
Nov 02 20:17:55.992538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:55.992431    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podStartSLOduration=1.9924166049999998 podStartE2EDuration="1.992416605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:55.988094778 +0000 UTC m=+1564.862365670" watchObservedRunningTime="2026-11-02 20:17:55.992416605 +0000 UTC m=+1564.866687492"
Nov 02 20:17:56.065663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:56.065627    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:56.066076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:56.066049    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:17:56.066271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:56.066245    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:17:56.728452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:56.728410    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:56.728894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:56.728476    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:57.085352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:57.085264    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:57.735355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:57.735312    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:57.735967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:57.735931    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:57.936145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:57.936110    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:57.953825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:57.946034    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:17:58.727977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:58.727940    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:58.728216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:58.727998    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:17:59.371092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.371052    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"]
Nov 02 20:17:59.375826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.375797    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d73d56c-n6bgd"]
Nov 02 20:17:59.447188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.447156    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:17:59.447471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.447423    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" containerID="cri-o://bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517" gracePeriod=30
Nov 02 20:17:59.737902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.732220    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 20:17:59.737902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:17:59.732293    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 20:18:00.834297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:00.831031    5099 patch_prober.go:29] interesting pod/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:18:00.834297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:00.831122    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:18:01.328337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:01.328298    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f71ca99-c569-41a0-8007-9437d191f6aa" path="/var/lib/kubelet/pods/0f71ca99-c569-41a0-8007-9437d191f6aa/volumes"
Nov 02 20:18:01.657147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:01.657114    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:18:01.659217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:01.659192    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:18:01.664081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:01.664028    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:18:02.751483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.749903    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:18:02.761671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.761643    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:18:02.761922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.761775    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:02.811872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.811843    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-55e81986\""
Nov 02 20:18:02.812009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.811872    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-55e81986-dockercfg-t2hht\""
Nov 02 20:18:02.882129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.882098    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc849\" (UniqueName: \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:02.882316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.882294    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:02.983666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.983626    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc849\" (UniqueName: \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:02.983915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.983775    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:02.984538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:02.984517    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:03.038007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:03.037932    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc849\" (UniqueName: \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\") pod \"payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:03.083794 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:03.083749    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:03.479303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:03.479250    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:18:03.805560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:03.805536    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:18:03.805936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:03.805600    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:18:04.116456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.116424    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" event={"ID":"30d12fc0-613c-401a-8ba2-48df651f627e","Type":"ContainerStarted","Data":"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"}
Nov 02 20:18:04.116658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.116479    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" event={"ID":"30d12fc0-613c-401a-8ba2-48df651f627e","Type":"ContainerStarted","Data":"580e3872bcfce7ecd00d2e37801431ab249c2c0c67cec25745d5ba0eee6d14d0"}
Nov 02 20:18:04.116658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.116505    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:04.226158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.226103    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" podStartSLOduration=2.226090384 podStartE2EDuration="2.226090384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:04.223148579 +0000 UTC m=+1573.097419468" watchObservedRunningTime="2026-11-02 20:18:04.226090384 +0000 UTC m=+1573.100361271"
Nov 02 20:18:04.775428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.775399    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:18:04.881461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881427    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881466    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881538    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881576    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881633    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881666    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881717    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881793    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:04.881997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.881945    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.882339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.882026    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\") pod \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\" (UID: \"11fa3d86-1d1e-400f-99d5-59f0833ca47a\") "
Nov 02 20:18:04.882339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.882047    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:04.882339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.882209    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:04.882491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.882463    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:04.882548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.882507    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:04.883110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.883086    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.883195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.883117    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.883195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.883135    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.883195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.883150    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.883195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.883164    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.885148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.885112    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:04.885272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.885215    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:04.885353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.885312    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:04.885526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.885504    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw" pod "11fa3d86-1d1e-400f-99d5-59f0833ca47a" (UID: "11fa3d86-1d1e-400f-99d5-59f0833ca47a"). InnerVolumeSpecName "kube-api-access-xhdkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:04.984838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.984801    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.984838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.984833    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdkw\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-kube-api-access-xhdkw\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.984838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.984844    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:04.985143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:04.984853    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fa3d86-1d1e-400f-99d5-59f0833ca47a-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:05.127757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.127722    5099 generic.go:356] "Generic (PLEG): container finished" podID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerID="bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517" exitCode=0
Nov 02 20:18:05.127942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.127776    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"
Nov 02 20:18:05.127942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.127835    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerDied","Data":"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"}
Nov 02 20:18:05.127942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.127903    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2" event={"ID":"11fa3d86-1d1e-400f-99d5-59f0833ca47a","Type":"ContainerDied","Data":"57a2adfbad12155a3c054c2829b19176510d77fa0865dd150f89c905fe4fc800"}
Nov 02 20:18:05.127942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.127931    5099 scope.go:122] "RemoveContainer" containerID="bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"
Nov 02 20:18:05.140489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.140462    5099 scope.go:122] "RemoveContainer" containerID="1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"
Nov 02 20:18:05.151247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.151224    5099 scope.go:122] "RemoveContainer" containerID="bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"
Nov 02 20:18:05.151533 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:18:05.151508    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517\": container with ID starting with bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517 not found: ID does not exist" containerID="bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"
Nov 02 20:18:05.151618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.151542    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517"} err="failed to get container status \"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517\": rpc error: code = NotFound desc = could not find container \"bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517\": container with ID starting with bf85b30bbceee19e37f6b5ca46ae487c4d050467b5be355a2f8da37dbb110517 not found: ID does not exist"
Nov 02 20:18:05.151618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.151561    5099 scope.go:122] "RemoveContainer" containerID="1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"
Nov 02 20:18:05.151850 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:18:05.151824    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7\": container with ID starting with 1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7 not found: ID does not exist" containerID="1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"
Nov 02 20:18:05.151902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.151861    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7"} err="failed to get container status \"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7\": rpc error: code = NotFound desc = could not find container \"1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7\": container with ID starting with 1bc469061d75fa43ba5a5c6c0db3fb9e1335287e50a1ca43572db48832b718b7 not found: ID does not exist"
Nov 02 20:18:05.291578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.291543    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:18:05.296457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.296431    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2d73d56c-openshift-default-6c6495954f-fbdx2"]
Nov 02 20:18:05.556632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.556540    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:18:05.557452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.557430    5099 cpu_manager.go:414] "containerMap: removing container" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:05.557452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.557452    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:05.557678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.557664    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:05.557746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.557693    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:05.563215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.563192    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595171    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595300    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595467    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595505    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595553    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hvs\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595693    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595731    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.599835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.595759    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.651799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.651742    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:18:05.698453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698417    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698490    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698553    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698658    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hvs\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698709    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698753    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.698880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698828    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.699048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698947    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.699048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.698993    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.699648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.699615    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.699913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.699889    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.700591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.700136    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.700808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.700765    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.700981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.700959    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.702061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.702031    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.703369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.703347    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.753137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.753109    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.753320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.753268    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hvs\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\") pod \"maas-default-gateway-openshift-default-696f47c6c-fpndb\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:05.879475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:05.879440    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:06.307171 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:18:06.307111    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4311b6c0_0384_40b5_b78f_9edee65e8051.slice/crio-f3016b69ebee7b2f89a3dd5cd6443eb30a013f46a3feed4f24908c1761ddda80 WatchSource:0}: Error finding container f3016b69ebee7b2f89a3dd5cd6443eb30a013f46a3feed4f24908c1761ddda80: Status 404 returned error can't find the container with id f3016b69ebee7b2f89a3dd5cd6443eb30a013f46a3feed4f24908c1761ddda80
Nov 02 20:18:06.308316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:06.308291    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:18:06.309885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:06.309857    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:18:06.309963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:06.309917    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:18:06.309963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:06.309948    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:18:07.176282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.176101    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" event={"ID":"4311b6c0-0384-40b5-b78f-9edee65e8051","Type":"ContainerStarted","Data":"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"}
Nov 02 20:18:07.176282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.176152    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" event={"ID":"4311b6c0-0384-40b5-b78f-9edee65e8051","Type":"ContainerStarted","Data":"f3016b69ebee7b2f89a3dd5cd6443eb30a013f46a3feed4f24908c1761ddda80"}
Nov 02 20:18:07.275749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.275716    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" path="/var/lib/kubelet/pods/11fa3d86-1d1e-400f-99d5-59f0833ca47a/volumes"
Nov 02 20:18:07.348815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.348621    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podStartSLOduration=2.348598287 podStartE2EDuration="2.348598287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:07.334676534 +0000 UTC m=+1576.208947423" watchObservedRunningTime="2026-11-02 20:18:07.348598287 +0000 UTC m=+1576.222869178"
Nov 02 20:18:07.879861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.879823    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:07.881145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.881112    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:07.881280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:07.881172    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:08.094151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.094115    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"]
Nov 02 20:18:08.094943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.094924    5099 cpu_manager.go:414] "containerMap: removing container" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:08.094943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.094941    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fa3d86-1d1e-400f-99d5-59f0833ca47a" containerName="istio-proxy"
Nov 02 20:18:08.099226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.099207    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.159856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.158997    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.159856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.159300    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hkzb\" (UniqueName: \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.159856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.159348    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.170572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.170541    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"]
Nov 02 20:18:08.201985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.201953    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:18:08.202172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.202031    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:18:08.267806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.266928    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hkzb\" (UniqueName: \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.267806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.267000    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.272808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.272241    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.272808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.272533    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.276110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.276080    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.325209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.325183    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hkzb\" (UniqueName: \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\") pod \"maas-api-revoke-sub-fed0efb2-cndnf\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.416029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.415955    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:08.828273 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:18:08.828240    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3062eb_3ca7_4dd1_92df_2176f38352cd.slice/crio-8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def WatchSource:0}: Error finding container 8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def: Status 404 returned error can't find the container with id 8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def
Nov 02 20:18:08.829919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.829892    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"]
Nov 02 20:18:08.880372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.880340    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:08.880478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:08.880396    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:09.070989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.070946    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"]
Nov 02 20:18:09.079292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.079161    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dcdf6a-v29xr"]
Nov 02 20:18:09.400021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.399983    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1feb72d-ca9a-4491-920f-59062f272917" path="/var/lib/kubelet/pods/c1feb72d-ca9a-4491-920f-59062f272917/volumes"
Nov 02 20:18:09.400746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.400720    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf" event={"ID":"ce3062eb-3ca7-4dd1-92df-2176f38352cd","Type":"ContainerStarted","Data":"8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def"}
Nov 02 20:18:09.894372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.894332    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:09.899890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:09.899848    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:10.094583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.090935    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:18:10.159967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.159933    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.258892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.227088    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-be1348\""
Nov 02 20:18:10.331811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.330093    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-be1348-dockercfg-lf42z\""
Nov 02 20:18:10.509105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.369795    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:18:10.559524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.559484    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn69n\" (UniqueName: \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.560048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.560013    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.672282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.672249    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn69n\" (UniqueName: \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.685800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.672589    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.686430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.676992    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.728810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.714106    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf" event={"ID":"ce3062eb-3ca7-4dd1-92df-2176f38352cd","Type":"ContainerStarted","Data":"49dc56a65acf8d341e00f39403037b59616046e713bf8a78b3420ec135f66f19"}
Nov 02 20:18:10.814497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.792529    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn69n\" (UniqueName: \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\") pod \"payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:10.889415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.889378    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:10.889625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.889435    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:10.896930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.896853    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf" podStartSLOduration=2.8968347530000003 podStartE2EDuration="2.896834753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:10.888221724 +0000 UTC m=+1579.762492614" watchObservedRunningTime="2026-11-02 20:18:10.896834753 +0000 UTC m=+1579.771105642"
Nov 02 20:18:10.979488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:10.979449    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:11.393715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.393681    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:18:11.748640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.748416    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" event={"ID":"79956486-4b68-4095-8d4d-1ef90c39fcfc","Type":"ContainerStarted","Data":"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"}
Nov 02 20:18:11.748640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.748490    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" event={"ID":"79956486-4b68-4095-8d4d-1ef90c39fcfc","Type":"ContainerStarted","Data":"65267b44e264be061be1f176db4fe75d579f5bbb9aae4c9b607ca2db9b06b39c"}
Nov 02 20:18:11.774613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.750149    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:11.804429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.803700    5099 generic.go:356] "Generic (PLEG): container finished" podID="ce3062eb-3ca7-4dd1-92df-2176f38352cd" containerID="49dc56a65acf8d341e00f39403037b59616046e713bf8a78b3420ec135f66f19" exitCode=0
Nov 02 20:18:11.804429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.804010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf" event={"ID":"ce3062eb-3ca7-4dd1-92df-2176f38352cd","Type":"ContainerDied","Data":"49dc56a65acf8d341e00f39403037b59616046e713bf8a78b3420ec135f66f19"}
Nov 02 20:18:11.877101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.877045    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" podStartSLOduration=2.877029704 podStartE2EDuration="2.877029704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:11.863523492 +0000 UTC m=+1580.737794382" watchObservedRunningTime="2026-11-02 20:18:11.877029704 +0000 UTC m=+1580.751300592"
Nov 02 20:18:11.880485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.880458    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:11.880646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:11.880509    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:12.880136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:12.880091    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:12.880623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:12.880196    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:13.091885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.091816    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:13.194240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.194209    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\") pod \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") "
Nov 02 20:18:13.194412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.194320    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\" (UniqueName: \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\") pod \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") "
Nov 02 20:18:13.194412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.194354    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\") pod \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\" (UID: \"ce3062eb-3ca7-4dd1-92df-2176f38352cd\") "
Nov 02 20:18:13.194820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.194759    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca" pod "ce3062eb-3ca7-4dd1-92df-2176f38352cd" (UID: "ce3062eb-3ca7-4dd1-92df-2176f38352cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:13.195708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.195682    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:13.196718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.196693    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca" pod "ce3062eb-3ca7-4dd1-92df-2176f38352cd" (UID: "ce3062eb-3ca7-4dd1-92df-2176f38352cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:18:13.196918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.196895    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb" pod "ce3062eb-3ca7-4dd1-92df-2176f38352cd" (UID: "ce3062eb-3ca7-4dd1-92df-2176f38352cd"). InnerVolumeSpecName "kube-api-access-8hkzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:13.297257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.297194    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3062eb-3ca7-4dd1-92df-2176f38352cd-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:13.297257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.297225    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hkzb\" (UniqueName: \"kubernetes.io/projected/ce3062eb-3ca7-4dd1-92df-2176f38352cd-kube-api-access-8hkzb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:13.900887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.897210    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:13.901475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.901421    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:13.928053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.928019    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"
Nov 02 20:18:13.932571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.932539    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf" event={"ID":"ce3062eb-3ca7-4dd1-92df-2176f38352cd","Type":"ContainerDied","Data":"8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def"}
Nov 02 20:18:13.932820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:13.932770    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e2e4b223dda2a4d25a0400a41953f5b7075a5b0d617c703278ace4ef0fd0def"
Nov 02 20:18:14.880219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:14.880177    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 20:18:14.880380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:14.880247    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 20:18:15.048652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:15.048613    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"]
Nov 02 20:18:15.096361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:15.096323    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ae6f89-2skkd"]
Nov 02 20:18:15.131913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:15.131841    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:18:15.299338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:15.299296    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caec533d-5e2d-4c29-8d60-1ab86165e48d" path="/var/lib/kubelet/pods/caec533d-5e2d-4c29-8d60-1ab86165e48d/volumes"
Nov 02 20:18:16.028629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:16.028587    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:16.029159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:16.029124    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:16.030682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:16.030659    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:18:16.098044 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:18:16.097995    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:18:18.850437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:18.850389    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:18:18.850978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:18.850610    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy" containerID="cri-o://1b74600168e074d10136f5e3a4fd78df7c8368eb2fc14439ae02cdef08fa6d55" gracePeriod=30
Nov 02 20:18:18.896091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:18.896053    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:18:18.896354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:18.896303    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy" containerID="cri-o://89222753b5e8a37eec5a8cb65aad9add555ec0328e49def9612301ae0cf17669" gracePeriod=30
Nov 02 20:18:23.930874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:23.930842    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:18:24.028195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.028154    5099 generic.go:356] "Generic (PLEG): container finished" podID="35c524ca-7874-48a5-82fc-7a5225208e08" containerID="1b74600168e074d10136f5e3a4fd78df7c8368eb2fc14439ae02cdef08fa6d55" exitCode=0
Nov 02 20:18:24.028351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.028333    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" event={"ID":"35c524ca-7874-48a5-82fc-7a5225208e08","Type":"ContainerDied","Data":"1b74600168e074d10136f5e3a4fd78df7c8368eb2fc14439ae02cdef08fa6d55"}
Nov 02 20:18:24.030270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.030246    5099 generic.go:356] "Generic (PLEG): container finished" podID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerID="89222753b5e8a37eec5a8cb65aad9add555ec0328e49def9612301ae0cf17669" exitCode=0
Nov 02 20:18:24.030401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.030318    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" event={"ID":"d4602a17-a939-4de6-9ab6-c3fc244164e0","Type":"ContainerDied","Data":"89222753b5e8a37eec5a8cb65aad9add555ec0328e49def9612301ae0cf17669"}
Nov 02 20:18:24.259236 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.259204    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:18:24.295168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.295144    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:18:24.313117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313082    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.313261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313159    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.313261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313208    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.313410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313376    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.313572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313544    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:24.313678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313546    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.313919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.313895    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.314107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314088    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.314193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314153    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.314250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314226    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.314350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314334    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.314427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314398    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\") pod \"35c524ca-7874-48a5-82fc-7a5225208e08\" (UID: \"35c524ca-7874-48a5-82fc-7a5225208e08\") "
Nov 02 20:18:24.314517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314493    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.314827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.314765    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.315865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.315388    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.315865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.315418    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.315865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.315435    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c524ca-7874-48a5-82fc-7a5225208e08-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.315865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.315450    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.315865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.315466    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.316553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.316466    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.316860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.316838    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:24.317795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.317761    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:24.318057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.318039    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz" pod "35c524ca-7874-48a5-82fc-7a5225208e08" (UID: "35c524ca-7874-48a5-82fc-7a5225208e08"). InnerVolumeSpecName "kube-api-access-qnvpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:24.416982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.416896    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.416982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.416932    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.416984    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417011    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417111    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417169    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417385    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417445    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.417507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417473    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\") pod \"d4602a17-a939-4de6-9ab6-c3fc244164e0\" (UID: \"d4602a17-a939-4de6-9ab6-c3fc244164e0\") "
Nov 02 20:18:24.418006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.417894    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.418260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418148    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:18:24.418485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418432    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418458    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c524ca-7874-48a5-82fc-7a5225208e08-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418473    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4602a17-a939-4de6-9ab6-c3fc244164e0-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418486    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418499    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c524ca-7874-48a5-82fc-7a5225208e08-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418513    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvpz\" (UniqueName: \"kubernetes.io/projected/35c524ca-7874-48a5-82fc-7a5225208e08-kube-api-access-qnvpz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.418620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418543    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.418836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418813    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.418894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.418877    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.420387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.420344    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:24.420626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.420590    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:18:24.421100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.421079    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:18:24.421209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.421180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v" pod "d4602a17-a939-4de6-9ab6-c3fc244164e0" (UID: "d4602a17-a939-4de6-9ab6-c3fc244164e0"). InnerVolumeSpecName "kube-api-access-qkw6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:18:24.520942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520899    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.520942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520939    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.521170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520953    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.521170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520967    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.521170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520983    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.521170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.520997    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkw6v\" (UniqueName: \"kubernetes.io/projected/d4602a17-a939-4de6-9ab6-c3fc244164e0-kube-api-access-qkw6v\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:24.521170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:24.521011    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4602a17-a939-4de6-9ab6-c3fc244164e0-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:18:25.040330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.040295    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"
Nov 02 20:18:25.040818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.040326    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj" event={"ID":"35c524ca-7874-48a5-82fc-7a5225208e08","Type":"ContainerDied","Data":"9588dea839494b84ed3dee6f51de47b4842dbfe5cfb9a2dd84628af1634ec922"}
Nov 02 20:18:25.040818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.040390    5099 scope.go:122] "RemoveContainer" containerID="1b74600168e074d10136f5e3a4fd78df7c8368eb2fc14439ae02cdef08fa6d55"
Nov 02 20:18:25.042068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.042047    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"
Nov 02 20:18:25.042211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.042180    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh" event={"ID":"d4602a17-a939-4de6-9ab6-c3fc244164e0","Type":"ContainerDied","Data":"7dcfb37554432af6940bad80e78be5db4d4f221eecccc8fbea8cfdd1b2af5665"}
Nov 02 20:18:25.054639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.054610    5099 scope.go:122] "RemoveContainer" containerID="89222753b5e8a37eec5a8cb65aad9add555ec0328e49def9612301ae0cf17669"
Nov 02 20:18:25.214086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.213990    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:18:25.259383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.259353    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-9xmsh"]
Nov 02 20:18:25.314559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.314521    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:18:25.323228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:25.323199    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-xtmtj"]
Nov 02 20:18:27.261915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:27.261881    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" path="/var/lib/kubelet/pods/35c524ca-7874-48a5-82fc-7a5225208e08/volumes"
Nov 02 20:18:27.262395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:27.262379    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" path="/var/lib/kubelet/pods/d4602a17-a939-4de6-9ab6-c3fc244164e0/volumes"
Nov 02 20:18:33.048584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:33.048547    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"]
Nov 02 20:18:33.094251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:33.094218    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f22ad2a-57mgc"]
Nov 02 20:18:33.262048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:33.262009    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2666e00-41fa-44df-a950-a264d69448f1" path="/var/lib/kubelet/pods/f2666e00-41fa-44df-a950-a264d69448f1/volumes"
Nov 02 20:18:43.045996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:43.045958    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"]
Nov 02 20:18:43.093008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:43.092972    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acd604b-gqkph"]
Nov 02 20:18:43.259547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:43.259509    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36b0c96-08b8-4bd6-a136-439bdbc8ba80" path="/var/lib/kubelet/pods/f36b0c96-08b8-4bd6-a136-439bdbc8ba80/volumes"
Nov 02 20:18:46.095289 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:18:46.095234    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:18:52.078752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:52.078710    5099 scope.go:122] "RemoveContainer" containerID="f65d9d91de66cb73bd58c39c3d1846094aa376995f74b1183946f78c1f72d1f4"
Nov 02 20:18:52.089595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:52.089572    5099 scope.go:122] "RemoveContainer" containerID="3cf3301ae0c045263865a8c6bba2763ff3a326ebc838c76049abc8093ae69806"
Nov 02 20:18:52.099291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:52.099270    5099 scope.go:122] "RemoveContainer" containerID="b9d79cae0ed4a27ea97064c32a8f51c2168a40db67f2548c0f9779f4a952725a"
Nov 02 20:18:52.109519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:52.109500    5099 scope.go:122] "RemoveContainer" containerID="95ed2762dcfc0bb46bd6ee4efce0adc0c9390c9411f01fbd95d0b9a8dc4e5b01"
Nov 02 20:18:54.046573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:54.046536    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"]
Nov 02 20:18:54.097004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:54.096961    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795afc2-hfzbf"]
Nov 02 20:18:55.260169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:55.260137    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bcd50b-ea1f-428f-8a1c-9975d613919d" path="/var/lib/kubelet/pods/52bcd50b-ea1f-428f-8a1c-9975d613919d/volumes"
Nov 02 20:18:56.045429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:56.045388    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"]
Nov 02 20:18:56.092429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:56.092389    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b506056e-7hw2z"]
Nov 02 20:18:57.261173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:18:57.261084    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67631a35-4b95-45bf-abcb-a9aec002d6ff" path="/var/lib/kubelet/pods/67631a35-4b95-45bf-abcb-a9aec002d6ff/volumes"
Nov 02 20:19:03.063153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.063113    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"]
Nov 02 20:19:03.106634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.106598    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-944488f9-l5gb5"]
Nov 02 20:19:03.261759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.261723    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e40349-5135-47f6-816f-bce368e018a5" path="/var/lib/kubelet/pods/e6e40349-5135-47f6-816f-bce368e018a5/volumes"
Nov 02 20:19:03.620287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.620254    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:19:03.621291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621259    5099 cpu_manager.go:414] "containerMap: removing container" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy"
Nov 02 20:19:03.621291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621290    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy"
Nov 02 20:19:03.621487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621336    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ce3062eb-3ca7-4dd1-92df-2176f38352cd" containerName="revoke-keys"
Nov 02 20:19:03.621487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621345    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3062eb-3ca7-4dd1-92df-2176f38352cd" containerName="revoke-keys"
Nov 02 20:19:03.621487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621388    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy"
Nov 02 20:19:03.621487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621398    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy"
Nov 02 20:19:03.621687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621571    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c524ca-7874-48a5-82fc-7a5225208e08" containerName="istio-proxy"
Nov 02 20:19:03.621687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621591    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3062eb-3ca7-4dd1-92df-2176f38352cd" containerName="revoke-keys"
Nov 02 20:19:03.621687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.621608    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4602a17-a939-4de6-9ab6-c3fc244164e0" containerName="istio-proxy"
Nov 02 20:19:03.627882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.627861    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.696753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.696725    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-602be312-openshift-default-dockercfg-4c2qs\""
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.718862    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlms\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.718997    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719043    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719102    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719173    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719206    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719348    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719466    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.720805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.719522    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.733129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.733095    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:19:03.825813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.824955    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.825813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.825055    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.825813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.825086    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.825813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.825228    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.833427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.829439    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.833427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.829731    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.883405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.883322    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.900235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.883699    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.915724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.915681    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.916492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.916465    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlms\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.922071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.922040    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.922638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.922618    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.923428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.922552    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.923583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.916290    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.936041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.935989    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.964802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.959176    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.992916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.992810    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlms\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:03.995478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:03.995448    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:04.259199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.259113    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:04.787365 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:19:04.787333    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1d3f3f_71f3_47ce_8eb2_bd9e271e5485.slice/crio-b006fd5cb66ea784b2f87d9d058082f24322ed0772f4f7cb8acc474ffd3958c6 WatchSource:0}: Error finding container b006fd5cb66ea784b2f87d9d058082f24322ed0772f4f7cb8acc474ffd3958c6: Status 404 returned error can't find the container with id b006fd5cb66ea784b2f87d9d058082f24322ed0772f4f7cb8acc474ffd3958c6
Nov 02 20:19:04.788973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.788949    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:19:04.790551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.790511    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:04.790626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.790587    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:04.790626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.790617    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:04.908191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.908158    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"]
Nov 02 20:19:04.913187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.913168    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:04.969669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.969438    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:19:04.969669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.969634    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:19:04.971568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:04.971523    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"]
Nov 02 20:19:05.016007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.015964    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.016263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.016232    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.016426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.016344    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqtm\" (UniqueName: \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.118302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.118274    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.118480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.118463    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.118548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.118507    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqtm\" (UniqueName: \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.119118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.119094    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.120809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.120771    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.174481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.174441    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqtm\" (UniqueName: \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\") pod \"maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.229397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.229362    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:05.617497 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:19:05.617464    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode333a509_b023_4f15_8bbc_ab43fc2daaa6.slice/crio-64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be WatchSource:0}: Error finding container 64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be: Status 404 returned error can't find the container with id 64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be
Nov 02 20:19:05.618915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.618889    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"]
Nov 02 20:19:05.661829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.661761    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerStarted","Data":"fe92a33b313e7cfb7934e78996f4b8107d0f4a2c03f974e81a4d9bacf04fba9f"}
Nov 02 20:19:05.661988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.661864    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerStarted","Data":"b006fd5cb66ea784b2f87d9d058082f24322ed0772f4f7cb8acc474ffd3958c6"}
Nov 02 20:19:05.663815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.663760    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj" event={"ID":"e333a509-b023-4f15-8bbc-ab43fc2daaa6","Type":"ContainerStarted","Data":"64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be"}
Nov 02 20:19:05.774938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:05.774875    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podStartSLOduration=2.774857459 podStartE2EDuration="2.774857459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:05.773367335 +0000 UTC m=+1634.647638223" watchObservedRunningTime="2026-11-02 20:19:05.774857459 +0000 UTC m=+1634.649128347"
Nov 02 20:19:06.259840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.259812    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:06.265686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.265662    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:06.683917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.683881    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj" event={"ID":"e333a509-b023-4f15-8bbc-ab43fc2daaa6","Type":"ContainerStarted","Data":"d450539bd95cb6e458fcf6e7e3a3ea322905deb0a0c1f27353691d4498cc6b81"}
Nov 02 20:19:06.684422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.684343    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:06.685493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.685470    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:19:06.793588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:06.793533    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj" podStartSLOduration=2.793517541 podStartE2EDuration="2.793517541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:06.791345471 +0000 UTC m=+1635.665616359" watchObservedRunningTime="2026-11-02 20:19:06.793517541 +0000 UTC m=+1635.667788429"
Nov 02 20:19:07.694421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:07.694385    5099 generic.go:356] "Generic (PLEG): container finished" podID="e333a509-b023-4f15-8bbc-ab43fc2daaa6" containerID="d450539bd95cb6e458fcf6e7e3a3ea322905deb0a0c1f27353691d4498cc6b81" exitCode=0
Nov 02 20:19:07.694864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:07.694472    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj" event={"ID":"e333a509-b023-4f15-8bbc-ab43fc2daaa6","Type":"ContainerDied","Data":"d450539bd95cb6e458fcf6e7e3a3ea322905deb0a0c1f27353691d4498cc6b81"}
Nov 02 20:19:08.889090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.889064    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:08.990600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.990521    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\") pod \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") "
Nov 02 20:19:08.990755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.990655    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\") pod \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") "
Nov 02 20:19:08.990825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.990806    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\" (UniqueName: \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\") pod \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\" (UID: \"e333a509-b023-4f15-8bbc-ab43fc2daaa6\") "
Nov 02 20:19:08.991047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.991023    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca" pod "e333a509-b023-4f15-8bbc-ab43fc2daaa6" (UID: "e333a509-b023-4f15-8bbc-ab43fc2daaa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:08.991462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.991445    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:08.992754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.992733    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca" pod "e333a509-b023-4f15-8bbc-ab43fc2daaa6" (UID: "e333a509-b023-4f15-8bbc-ab43fc2daaa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:19:08.992881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:08.992859    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm" pod "e333a509-b023-4f15-8bbc-ab43fc2daaa6" (UID: "e333a509-b023-4f15-8bbc-ab43fc2daaa6"). InnerVolumeSpecName "kube-api-access-2gqtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:09.092962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:09.092929    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e333a509-b023-4f15-8bbc-ab43fc2daaa6-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.092962 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:09.092957    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqtm\" (UniqueName: \"kubernetes.io/projected/e333a509-b023-4f15-8bbc-ab43fc2daaa6-kube-api-access-2gqtm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:09.712062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:09.712025    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"
Nov 02 20:19:09.712062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:09.712040    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj" event={"ID":"e333a509-b023-4f15-8bbc-ab43fc2daaa6","Type":"ContainerDied","Data":"64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be"}
Nov 02 20:19:09.712270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:09.712092    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ed3ee432fac3343ed4817abd918cf1f94e45e554a7e673696b8ae77b4c08be"
Nov 02 20:19:13.498843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.498806    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:19:13.506708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.506506    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e333a509-b023-4f15-8bbc-ab43fc2daaa6" containerName="revoke-keys"
Nov 02 20:19:13.506708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.506538    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e333a509-b023-4f15-8bbc-ab43fc2daaa6" containerName="revoke-keys"
Nov 02 20:19:13.506995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.506977    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e333a509-b023-4f15-8bbc-ab43fc2daaa6" containerName="revoke-keys"
Nov 02 20:19:13.526830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.515464    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:19:13.526830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.515533    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:19:13.526830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.516981    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.526830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.526100    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:19:13.526830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.526193    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.562283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.562248    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.562539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.562520    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.567575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.567551    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-602be312\""
Nov 02 20:19:13.575405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.575381    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-602be312-dockercfg-5whzk\""
Nov 02 20:19:13.682810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.665065    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.682810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.665254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvxb\" (UniqueName: \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.682810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.665296    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.682810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.665351    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.682810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.666707    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:13.766944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.766856    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvxb\" (UniqueName: \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.767519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.767490    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.768907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.768883    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.892804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.881931    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvxb\" (UniqueName: \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:13.892804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:13.882424    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:14.057075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:14.056977    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"]
Nov 02 20:19:14.061879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:14.061845    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1422d13d-2cpx6"]
Nov 02 20:19:14.145208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:14.145172    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:14.154490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:14.154462    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:15.005844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:15.005671    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:19:15.021757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:15.021680    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:19:15.025567 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:19:15.025127    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb133cdf1_88c5_4a41_8619_10ef0aaa874b.slice/crio-78486fa38c0ee31c72b60355d354e1869cea347f5bf8e0ed8bfe0955f565ac35 WatchSource:0}: Error finding container 78486fa38c0ee31c72b60355d354e1869cea347f5bf8e0ed8bfe0955f565ac35: Status 404 returned error can't find the container with id 78486fa38c0ee31c72b60355d354e1869cea347f5bf8e0ed8bfe0955f565ac35
Nov 02 20:19:15.146460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:15.139883    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" event={"ID":"6e00f63b-f05c-4e01-b83b-5753096a40cb","Type":"ContainerStarted","Data":"2d43b1ca64af915d55a784d703ef0fdd9b1e70659233473eabc37d9fbe93e81e"}
Nov 02 20:19:15.146460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:15.142186    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" event={"ID":"b133cdf1-88c5-4a41-8619-10ef0aaa874b","Type":"ContainerStarted","Data":"78486fa38c0ee31c72b60355d354e1869cea347f5bf8e0ed8bfe0955f565ac35"}
Nov 02 20:19:15.298644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:15.298610    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0be176b-40bd-43db-9f3a-2078822aa931" path="/var/lib/kubelet/pods/f0be176b-40bd-43db-9f3a-2078822aa931/volumes"
Nov 02 20:19:16.379121 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:16.379078    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:19:16.415815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.411741    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" event={"ID":"b133cdf1-88c5-4a41-8619-10ef0aaa874b","Type":"ContainerStarted","Data":"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"}
Nov 02 20:19:16.416347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.416326    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:16.431913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.431868    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:16.477807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.471401    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" event={"ID":"6e00f63b-f05c-4e01-b83b-5753096a40cb","Type":"ContainerStarted","Data":"e94aa3f8f8f9f2f10d393b40176bc7f77f09e52e9259b534eb1507a93eb3d075"}
Nov 02 20:19:16.477807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.472373    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:16.548818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.548670    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" podStartSLOduration=3.548649373 podStartE2EDuration="3.548649373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:16.548042639 +0000 UTC m=+1645.422313528" watchObservedRunningTime="2026-11-02 20:19:16.548649373 +0000 UTC m=+1645.422920262"
Nov 02 20:19:16.673875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.668016    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" podStartSLOduration=3.667996482 podStartE2EDuration="3.667996482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:16.665181791 +0000 UTC m=+1645.539452679" watchObservedRunningTime="2026-11-02 20:19:16.667996482 +0000 UTC m=+1645.542267371"
Nov 02 20:19:16.757230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.756883    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-htsgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:19:16.757230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.756948    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:16.878807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:16.864004    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:19:17.078273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.077880    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-fpndb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:19:17.078273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.077949    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:17.105936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.105886    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:17.183946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.183885    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:17.306599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.306552    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:19:17.383879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.381720    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:17.745469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.741531    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-gmgt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:19:17.745469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:17.741671    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:19:21.879466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:21.879409    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" podUID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" containerName="payload-pre-processing" containerID="cri-o://9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7" gracePeriod=25
Nov 02 20:19:22.209892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.209857    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:19:22.356996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.356958    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\" (UniqueName: \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\") pod \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") "
Nov 02 20:19:22.357307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.357288    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\") pod \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\" (UID: \"9136f10a-994a-4b13-bfb1-50bd5bee2ac1\") "
Nov 02 20:19:22.357724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.357693    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume" pod "9136f10a-994a-4b13-bfb1-50bd5bee2ac1" (UID: "9136f10a-994a-4b13-bfb1-50bd5bee2ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:22.362324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.362299    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97" pod "9136f10a-994a-4b13-bfb1-50bd5bee2ac1" (UID: "9136f10a-994a-4b13-bfb1-50bd5bee2ac1"). InnerVolumeSpecName "kube-api-access-gpp97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:22.459686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.459602    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpp97\" (UniqueName: \"kubernetes.io/projected/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-kube-api-access-gpp97\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:22.459686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.459632    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9136f10a-994a-4b13-bfb1-50bd5bee2ac1-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:22.895667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.895635    5099 generic.go:356] "Generic (PLEG): container finished" podID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" containerID="9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7" exitCode=0
Nov 02 20:19:22.896124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.895678    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"
Nov 02 20:19:22.896124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.895716    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" event={"ID":"9136f10a-994a-4b13-bfb1-50bd5bee2ac1","Type":"ContainerDied","Data":"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"}
Nov 02 20:19:22.896124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.895768    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4" event={"ID":"9136f10a-994a-4b13-bfb1-50bd5bee2ac1","Type":"ContainerDied","Data":"b8daa51bcff16adf9404aad79c74e6107acdadcbb2b4416bdfa4939937046eba"}
Nov 02 20:19:22.896124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.895801    5099 scope.go:122] "RemoveContainer" containerID="9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"
Nov 02 20:19:22.907956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.907939    5099 scope.go:122] "RemoveContainer" containerID="9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"
Nov 02 20:19:22.908266 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:22.908246    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7\": container with ID starting with 9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7 not found: ID does not exist" containerID="9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"
Nov 02 20:19:22.908328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:22.908276    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7"} err="failed to get container status \"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7\": rpc error: code = NotFound desc = could not find container \"9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7\": container with ID starting with 9d106500a87f056c32f72b02b22623ba62985d1757077543810945e0cb75cca7 not found: ID does not exist"
Nov 02 20:19:23.050064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.050028    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"]
Nov 02 20:19:23.054674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.054645    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:19:23.096791 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.096753    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f16f285-59z8w"]
Nov 02 20:19:23.101170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.101145    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f75722a-57595949f9-tq9w4"]
Nov 02 20:19:23.297905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.297795    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55875403-32d0-4ac0-a81c-08ea028493a2" path="/var/lib/kubelet/pods/55875403-32d0-4ac0-a81c-08ea028493a2/volumes"
Nov 02 20:19:23.298275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:23.298259    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" path="/var/lib/kubelet/pods/9136f10a-994a-4b13-bfb1-50bd5bee2ac1/volumes"
Nov 02 20:19:28.059558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:28.059495    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"]
Nov 02 20:19:28.114803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:28.114053    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dad84ad-dkjcl"]
Nov 02 20:19:28.870640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:28.870605    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:19:28.870826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:28.870744    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:19:29.262022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:29.261933    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92764572-a2b5-4a56-a0f2-b9195609440c" path="/var/lib/kubelet/pods/92764572-a2b5-4a56-a0f2-b9195609440c/volumes"
Nov 02 20:19:34.091843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:34.091706    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:19:34.092369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:34.092011    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy" containerID="cri-o://04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe" gracePeriod=30
Nov 02 20:19:35.758545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.758504    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"]
Nov 02 20:19:35.759430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.759410    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" containerName="payload-pre-processing"
Nov 02 20:19:35.759483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.759432    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" containerName="payload-pre-processing"
Nov 02 20:19:35.759559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.759547    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9136f10a-994a-4b13-bfb1-50bd5bee2ac1" containerName="payload-pre-processing"
Nov 02 20:19:35.764514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.764495    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.815697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.815661    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"]
Nov 02 20:19:35.840980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.840943    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.841155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.841075    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.841345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.841328    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8qh\" (UniqueName: \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.856378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.856350    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:19:35.856541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.856461    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:19:35.942900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.942865    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.943083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.942938    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.943197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.943179    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8qh\" (UniqueName: \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.943670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.943642    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.945293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.945266    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:35.998070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:35.998031    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8qh\" (UniqueName: \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\") pod \"maas-api-revoke-keys-e2e-ait-13c1d955-dh25t\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:36.080325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:36.080245    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:36.463631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:36.463590    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"]
Nov 02 20:19:37.095012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:37.094922    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t" event={"ID":"b688f777-3b77-4d12-8dd7-534ea472cece","Type":"ContainerStarted","Data":"84d9b9b5c47a16505852b8a06f0c7545ba5b53da0149a64e552a87737b6f7b71"}
Nov 02 20:19:37.095012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:37.094983    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t" event={"ID":"b688f777-3b77-4d12-8dd7-534ea472cece","Type":"ContainerStarted","Data":"4a8761ffe3402be4bde8e7cf463837ddf35c878f5b2d4a166c4f4cab50942b64"}
Nov 02 20:19:37.208825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:37.208712    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t" podStartSLOduration=2.208695949 podStartE2EDuration="2.208695949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:37.201930328 +0000 UTC m=+1666.076201218" watchObservedRunningTime="2026-11-02 20:19:37.208695949 +0000 UTC m=+1666.082966838"
Nov 02 20:19:38.105808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:38.105713    5099 generic.go:356] "Generic (PLEG): container finished" podID="b688f777-3b77-4d12-8dd7-534ea472cece" containerID="84d9b9b5c47a16505852b8a06f0c7545ba5b53da0149a64e552a87737b6f7b71" exitCode=0
Nov 02 20:19:38.106216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:38.105836    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t" event={"ID":"b688f777-3b77-4d12-8dd7-534ea472cece","Type":"ContainerDied","Data":"84d9b9b5c47a16505852b8a06f0c7545ba5b53da0149a64e552a87737b6f7b71"}
Nov 02 20:19:39.335365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.335327    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:39.420858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.420834    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\") pod \"b688f777-3b77-4d12-8dd7-534ea472cece\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") "
Nov 02 20:19:39.421016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.420911    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\") pod \"b688f777-3b77-4d12-8dd7-534ea472cece\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") "
Nov 02 20:19:39.421016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.420987    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\" (UniqueName: \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\") pod \"b688f777-3b77-4d12-8dd7-534ea472cece\" (UID: \"b688f777-3b77-4d12-8dd7-534ea472cece\") "
Nov 02 20:19:39.421313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.421267    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca" pod "b688f777-3b77-4d12-8dd7-534ea472cece" (UID: "b688f777-3b77-4d12-8dd7-534ea472cece"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:39.422099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.422070    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.423411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.423377    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca" pod "b688f777-3b77-4d12-8dd7-534ea472cece" (UID: "b688f777-3b77-4d12-8dd7-534ea472cece"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:19:39.423613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.423595    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh" pod "b688f777-3b77-4d12-8dd7-534ea472cece" (UID: "b688f777-3b77-4d12-8dd7-534ea472cece"). InnerVolumeSpecName "kube-api-access-cf8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:39.427128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.427106    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:19:39.522813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.522764    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.522996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.522862    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.522996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.522900    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.522996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.522937    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523060    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:39.523229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523079    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523129    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523168    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523200    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523252    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\") pod \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\" (UID: \"6068698f-2e9f-4167-94be-ba4a3e8a78ee\") "
Nov 02 20:19:39.523505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523325    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:39.523505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523342    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:39.523505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523448    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:39.523715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.523611    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:39.524111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524085    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524116    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b688f777-3b77-4d12-8dd7-534ea472cece-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524131    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf8qh\" (UniqueName: \"kubernetes.io/projected/b688f777-3b77-4d12-8dd7-534ea472cece-kube-api-access-cf8qh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524148    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524163    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524177    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.524205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.524191    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.525532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.525502    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:39.525625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.525547    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:19:39.525815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.525777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:19:39.525815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.525803    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6" pod "6068698f-2e9f-4167-94be-ba4a3e8a78ee" (UID: "6068698f-2e9f-4167-94be-ba4a3e8a78ee"). InnerVolumeSpecName "kube-api-access-9r7j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:39.625888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.625843    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.625888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.625882    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7j6\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-kube-api-access-9r7j6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.625888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.625897    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:39.626137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:39.625909    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6068698f-2e9f-4167-94be-ba4a3e8a78ee-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:40.126545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.126507    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"
Nov 02 20:19:40.126736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.126547    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t" event={"ID":"b688f777-3b77-4d12-8dd7-534ea472cece","Type":"ContainerDied","Data":"4a8761ffe3402be4bde8e7cf463837ddf35c878f5b2d4a166c4f4cab50942b64"}
Nov 02 20:19:40.126736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.126606    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8761ffe3402be4bde8e7cf463837ddf35c878f5b2d4a166c4f4cab50942b64"
Nov 02 20:19:40.128266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.128239    5099 generic.go:356] "Generic (PLEG): container finished" podID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerID="04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe" exitCode=0
Nov 02 20:19:40.128393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.128285    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"
Nov 02 20:19:40.128393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.128363    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" event={"ID":"6068698f-2e9f-4167-94be-ba4a3e8a78ee","Type":"ContainerDied","Data":"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"}
Nov 02 20:19:40.128501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.128412    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq" event={"ID":"6068698f-2e9f-4167-94be-ba4a3e8a78ee","Type":"ContainerDied","Data":"74ff60855871c128addb18cf90904e34a56abaac3c00eecd7723e2902df56cff"}
Nov 02 20:19:40.128501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.128444    5099 scope.go:122] "RemoveContainer" containerID="04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"
Nov 02 20:19:40.152074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.152045    5099 scope.go:122] "RemoveContainer" containerID="04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"
Nov 02 20:19:40.152480 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:40.152449    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe\": container with ID starting with 04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe not found: ID does not exist" containerID="04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"
Nov 02 20:19:40.152573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.152491    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe"} err="failed to get container status \"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe\": rpc error: code = NotFound desc = could not find container \"04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe\": container with ID starting with 04cb4eb89eaad9951dec229c3cfe526454cc790d4fa14f8a03b13719e6462cfe not found: ID does not exist"
Nov 02 20:19:40.334335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.334305    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:19:40.338931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:40.338903    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-qlsrq"]
Nov 02 20:19:41.047710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:41.047632    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"]
Nov 02 20:19:41.053436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:41.053087    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f397d02e-pp826"]
Nov 02 20:19:41.261147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:41.261116    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" path="/var/lib/kubelet/pods/6068698f-2e9f-4167-94be-ba4a3e8a78ee/volumes"
Nov 02 20:19:41.261628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:41.261613    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f4a7ea-2577-4983-9ae4-0f4209c34e9d" path="/var/lib/kubelet/pods/f3f4a7ea-2577-4983-9ae4-0f4209c34e9d/volumes"
Nov 02 20:19:43.046618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.046584    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"]
Nov 02 20:19:43.093704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.093671    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"]
Nov 02 20:19:43.097976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.097959    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb40bd-n2ggr"]
Nov 02 20:19:43.104096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.104065    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42466ff3-phc89"]
Nov 02 20:19:43.260852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.260798    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2152e865-e5c9-4e2a-995f-ce1eb7044d2a" path="/var/lib/kubelet/pods/2152e865-e5c9-4e2a-995f-ce1eb7044d2a/volumes"
Nov 02 20:19:43.261369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:43.261352    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e563929-5245-4353-a999-6271c9309709" path="/var/lib/kubelet/pods/5e563929-5245-4353-a999-6271c9309709/volumes"
Nov 02 20:19:46.098574 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:46.098533    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:19:47.044628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:47.044590    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"]
Nov 02 20:19:47.091377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:47.091340    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07182027-9n5rm"]
Nov 02 20:19:47.280252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:47.280217    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f5e60a-99fc-45ff-97ce-1949bacd9159" path="/var/lib/kubelet/pods/83f5e60a-99fc-45ff-97ce-1949bacd9159/volumes"
Nov 02 20:19:49.792894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:49.792849    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:19:49.917224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:49.917139    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.546848    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"]
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548154    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b688f777-3b77-4d12-8dd7-534ea472cece" containerName="revoke-keys"
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548180    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b688f777-3b77-4d12-8dd7-534ea472cece" containerName="revoke-keys"
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548215    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy"
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548224    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy"
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548503    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b688f777-3b77-4d12-8dd7-534ea472cece" containerName="revoke-keys"
Nov 02 20:19:51.549864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.548520    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6068698f-2e9f-4167-94be-ba4a3e8a78ee" containerName="istio-proxy"
Nov 02 20:19:51.601890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.584462    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"]
Nov 02 20:19:51.601890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.584595    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.694103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.671603    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.694103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.671653    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.694103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.671861    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkvwb\" (UniqueName: \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.694103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.692026    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:19:51.694103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.692101    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:19:51.799817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.789113    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.800163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.800097    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.818807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.810167    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvwb\" (UniqueName: \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:51.989837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:51.791665    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:52.024518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.024482    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:52.231920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.231888    5099 scope.go:122] "RemoveContainer" containerID="013a17dd1dbd2044902e7da93e17e31089f88b9e25654ad89640e9c3f689ad65"
Nov 02 20:19:52.290240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.290198    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvwb\" (UniqueName: \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\") pod \"maas-api-revoke-keys-e2e-ait-55e81986-ckrwm\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:52.293485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.293452    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:52.424807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.421945    5099 scope.go:122] "RemoveContainer" containerID="e7ffd72cda64524e9c82b1d0ca7c7029c9ec0b448b056ddb25acc5e2b08eceab"
Nov 02 20:19:52.602005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.595195    5099 scope.go:122] "RemoveContainer" containerID="0a2b2385f6a76fabf5fbb9831c38c3b096fc17257f958a75eabdb9965747c7df"
Nov 02 20:19:52.616680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.614918    5099 scope.go:122] "RemoveContainer" containerID="775e4078b1b20cb3e9c11a220c2863b3feca8478f4eed8979e2f2913d0cbfa6b"
Nov 02 20:19:52.635714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.635343    5099 scope.go:122] "RemoveContainer" containerID="9eec75596014a1b71fa98b0f02e4f7ee3bae6ed747c6e1051e475b1015868c46"
Nov 02 20:19:52.885987 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:19:52.885706    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d39233_093c_454e_8952_8c81f1839c32.slice/crio-68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71 WatchSource:0}: Error finding container 68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71: Status 404 returned error can't find the container with id 68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71
Nov 02 20:19:52.889393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:52.889361    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"]
Nov 02 20:19:53.542698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:53.542597    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm" event={"ID":"85d39233-093c-454e-8952-8c81f1839c32","Type":"ContainerStarted","Data":"ad281fd0b55213ae5da4c8fa17491c564420e9f6f5c3936d0aa0ca8b9a7d8da0"}
Nov 02 20:19:53.542698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:53.542659    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm" event={"ID":"85d39233-093c-454e-8952-8c81f1839c32","Type":"ContainerStarted","Data":"68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71"}
Nov 02 20:19:53.656289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:53.656233    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm" podStartSLOduration=2.656218522 podStartE2EDuration="2.656218522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:53.651649984 +0000 UTC m=+1682.525920874" watchObservedRunningTime="2026-11-02 20:19:53.656218522 +0000 UTC m=+1682.530489410"
Nov 02 20:19:54.553923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:54.553892    5099 generic.go:356] "Generic (PLEG): container finished" podID="85d39233-093c-454e-8952-8c81f1839c32" containerID="ad281fd0b55213ae5da4c8fa17491c564420e9f6f5c3936d0aa0ca8b9a7d8da0" exitCode=0
Nov 02 20:19:54.554107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:54.553988    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm" event={"ID":"85d39233-093c-454e-8952-8c81f1839c32","Type":"ContainerDied","Data":"ad281fd0b55213ae5da4c8fa17491c564420e9f6f5c3936d0aa0ca8b9a7d8da0"}
Nov 02 20:19:54.800163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:54.800110    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" podUID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" containerName="payload-processing" containerID="cri-o://adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9" gracePeriod=25
Nov 02 20:19:54.918028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:54.917994    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" podUID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" containerName="payload-pre-processing" containerID="cri-o://3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825" gracePeriod=25
Nov 02 20:19:55.122544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.122514    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:19:55.145645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.135843    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\") pod \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") "
Nov 02 20:19:55.145645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.135986    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\") pod \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\" (UID: \"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4\") "
Nov 02 20:19:55.145645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.136316    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume" pod "6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" (UID: "6d929d3f-2f5b-48fe-b2e6-e843e05d73b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:55.145645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.136853    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:55.145645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.139192    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz" pod "6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" (UID: "6d929d3f-2f5b-48fe-b2e6-e843e05d73b4"). InnerVolumeSpecName "kube-api-access-bqtvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:55.313320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.311855    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4-kube-api-access-bqtvz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:55.583812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.572753    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:19:55.617673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.617637    5099 generic.go:356] "Generic (PLEG): container finished" podID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" containerID="3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825" exitCode=0
Nov 02 20:19:55.617673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.617679    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"
Nov 02 20:19:55.617949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.617801    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" event={"ID":"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73","Type":"ContainerDied","Data":"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"}
Nov 02 20:19:55.617949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.617857    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk" event={"ID":"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73","Type":"ContainerDied","Data":"c5f29da348328ea6fbcc3d2c33a3b3d9d19a7093964b19e4316a126ad9173791"}
Nov 02 20:19:55.617949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.617889    5099 scope.go:122] "RemoveContainer" containerID="3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"
Nov 02 20:19:55.620764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.620735    5099 generic.go:356] "Generic (PLEG): container finished" podID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" containerID="adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9" exitCode=0
Nov 02 20:19:55.620921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.620773    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"
Nov 02 20:19:55.620921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.620881    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" event={"ID":"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4","Type":"ContainerDied","Data":"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"}
Nov 02 20:19:55.621044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.620953    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc" event={"ID":"6d929d3f-2f5b-48fe-b2e6-e843e05d73b4","Type":"ContainerDied","Data":"d615654cc8a18659d5540ca27ecd2480628cf20f0dc08c9d499908726f5afbeb"}
Nov 02 20:19:55.634007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.633691    5099 scope.go:122] "RemoveContainer" containerID="3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"
Nov 02 20:19:55.634246 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:55.634216    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825\": container with ID starting with 3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825 not found: ID does not exist" containerID="3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"
Nov 02 20:19:55.634365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.634255    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825"} err="failed to get container status \"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825\": rpc error: code = NotFound desc = could not find container \"3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825\": container with ID starting with 3b5e0688a05c81763ae5bd29f0bb2d9dde75baaea3dafeabd4933c233cfa2825 not found: ID does not exist"
Nov 02 20:19:55.634365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.634278    5099 scope.go:122] "RemoveContainer" containerID="adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"
Nov 02 20:19:55.652975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.652823    5099 scope.go:122] "RemoveContainer" containerID="adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"
Nov 02 20:19:55.654576 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:19:55.654500    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9\": container with ID starting with adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9 not found: ID does not exist" containerID="adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"
Nov 02 20:19:55.654576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.654534    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9"} err="failed to get container status \"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9\": rpc error: code = NotFound desc = could not find container \"adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9\": container with ID starting with adcf0a4abff3af0e4ae058bfb6873cd63cf4c71ae8cf229363e8a78f19d387c9 not found: ID does not exist"
Nov 02 20:19:55.730679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.730633    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\") pod \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") "
Nov 02 20:19:55.730893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.730873    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\" (UniqueName: \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\") pod \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\" (UID: \"acdcd4d9-91f7-4303-bbc3-7bda3aaafd73\") "
Nov 02 20:19:55.731203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.731130    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume" pod "acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" (UID: "acdcd4d9-91f7-4303-bbc3-7bda3aaafd73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:55.731699 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.731672    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:55.733769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.733732    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d" pod "acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" (UID: "acdcd4d9-91f7-4303-bbc3-7bda3aaafd73"). InnerVolumeSpecName "kube-api-access-6729d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:55.823152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.823126    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:55.833133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.833101    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6729d\" (UniqueName: \"kubernetes.io/projected/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73-kube-api-access-6729d\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:55.846324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.845516    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:19:55.859366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.859328    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13c1d955-6898b979c7-7jmlc"]
Nov 02 20:19:55.934424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.934397    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\") pod \"85d39233-093c-454e-8952-8c81f1839c32\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") "
Nov 02 20:19:55.934600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.934495    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\") pod \"85d39233-093c-454e-8952-8c81f1839c32\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") "
Nov 02 20:19:55.934600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.934544    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\" (UniqueName: \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\") pod \"85d39233-093c-454e-8952-8c81f1839c32\" (UID: \"85d39233-093c-454e-8952-8c81f1839c32\") "
Nov 02 20:19:55.934875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.934845    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca" pod "85d39233-093c-454e-8952-8c81f1839c32" (UID: "85d39233-093c-454e-8952-8c81f1839c32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:19:55.935344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.935321    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d39233-093c-454e-8952-8c81f1839c32-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:55.936653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.936628    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca" pod "85d39233-093c-454e-8952-8c81f1839c32" (UID: "85d39233-093c-454e-8952-8c81f1839c32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:19:55.936920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:55.936903    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb" pod "85d39233-093c-454e-8952-8c81f1839c32" (UID: "85d39233-093c-454e-8952-8c81f1839c32"). InnerVolumeSpecName "kube-api-access-tkvwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:19:56.036763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.036717    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d39233-093c-454e-8952-8c81f1839c32-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:56.036763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.036766    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkvwb\" (UniqueName: \"kubernetes.io/projected/85d39233-093c-454e-8952-8c81f1839c32-kube-api-access-tkvwb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:19:56.055415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.055367    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:19:56.059910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.059882    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-13c1d955-756b7ddfcf-ghwgk"]
Nov 02 20:19:56.634644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.634613    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm" event={"ID":"85d39233-093c-454e-8952-8c81f1839c32","Type":"ContainerDied","Data":"68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71"}
Nov 02 20:19:56.634850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.634656    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"
Nov 02 20:19:56.634850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:56.634661    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68702ea1f032075e53231678bacda76ede812373fd273bae1ec40dfc78600b71"
Nov 02 20:19:57.046887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.046847    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"]
Nov 02 20:19:57.051907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.051878    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb04643-bh2dc"]
Nov 02 20:19:57.263366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.263291    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" path="/var/lib/kubelet/pods/6d929d3f-2f5b-48fe-b2e6-e843e05d73b4/volumes"
Nov 02 20:19:57.263845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.263820    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" path="/var/lib/kubelet/pods/acdcd4d9-91f7-4303-bbc3-7bda3aaafd73/volumes"
Nov 02 20:19:57.264418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.264394    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6f89fd-4c0f-461a-9060-129801d6bafd" path="/var/lib/kubelet/pods/fa6f89fd-4c0f-461a-9060-129801d6bafd/volumes"
Nov 02 20:19:57.937209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.937176    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:19:57.937538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:57.937239    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:19:58.047373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.047292    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"]
Nov 02 20:19:58.094142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.094106    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70c1ccd-rw62r"]
Nov 02 20:19:58.670220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.670179    5099 generic.go:356] "Generic (PLEG): container finished" podID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerID="0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8" exitCode=0
Nov 02 20:19:58.670429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.670250    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerDied","Data":"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"}
Nov 02 20:19:58.670766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.670747    5099 scope.go:122] "RemoveContainer" containerID="0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"
Nov 02 20:19:58.672533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.672504    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:58.672632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.672570    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:58.672632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:58.672602    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:19:59.270308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:59.270273    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850586d8-47f1-45e0-b978-340a6dba4292" path="/var/lib/kubelet/pods/850586d8-47f1-45e0-b978-340a6dba4292/volumes"
Nov 02 20:19:59.683412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:19:59.683365    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerStarted","Data":"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"}
Nov 02 20:20:00.068309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:00.068212    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:00.068500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:00.068345    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:00.068500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:00.068405    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:01.069458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:01.069424    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:01.070241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:01.070206    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:02.065620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:02.065582    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:02.065893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:02.065865    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:03.061125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:03.061084    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:03.061731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:03.061697    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:04.068301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.068265    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:04.068937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.068900    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:04.531188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.531147    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:20:04.687872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.687839    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:20:04.688859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.688836    5099 cpu_manager.go:414] "containerMap: removing container" podUID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" containerName="payload-pre-processing"
Nov 02 20:20:04.688984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.688974    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" containerName="payload-pre-processing"
Nov 02 20:20:04.689063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689056    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" containerName="payload-processing"
Nov 02 20:20:04.689113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689106    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" containerName="payload-processing"
Nov 02 20:20:04.689172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689165    5099 cpu_manager.go:414] "containerMap: removing container" podUID="85d39233-093c-454e-8952-8c81f1839c32" containerName="revoke-keys"
Nov 02 20:20:04.689219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689212    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d39233-093c-454e-8952-8c81f1839c32" containerName="revoke-keys"
Nov 02 20:20:04.689424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689412    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdcd4d9-91f7-4303-bbc3-7bda3aaafd73" containerName="payload-pre-processing"
Nov 02 20:20:04.689502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689495    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d929d3f-2f5b-48fe-b2e6-e843e05d73b4" containerName="payload-processing"
Nov 02 20:20:04.689560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.689554    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d39233-093c-454e-8952-8c81f1839c32" containerName="revoke-keys"
Nov 02 20:20:04.692486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.692419    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.754809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.744992    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a51b3f-openshift-default-dockercfg-9q5l2\""
Nov 02 20:20:04.792486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.792367    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:20:04.826522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826484    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826563    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826606    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826653    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826689    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx24\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826706    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826729    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.826764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826755    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:04.827343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.826848    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.932514    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.932588    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.932802    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.932885    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.932942    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.933015    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.933067    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx24\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.933091    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.046920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:04.940113    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.048593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.048159    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.056471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.056430    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.057647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.057618    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.058276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.058252    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.058746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.058695    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.064987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.064951    5099 patch_prober.go:29] interesting pod/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 02 20:20:05.117410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.117224    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx24\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.121400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.121365    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.121824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.121719    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 02 20:20:05.139693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.131113    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.145535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.145470    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\") pod \"e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") " pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.354854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.313049    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:05.975255 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:05.975216    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924b5181_cabf_49bf_98aa_e6b4e0d87a1d.slice/crio-d29710e6f63e73befb580df438a5bfa2f4754ea1c8e35b06c6d001a38fe75037 WatchSource:0}: Error finding container d29710e6f63e73befb580df438a5bfa2f4754ea1c8e35b06c6d001a38fe75037: Status 404 returned error can't find the container with id d29710e6f63e73befb580df438a5bfa2f4754ea1c8e35b06c6d001a38fe75037
Nov 02 20:20:05.975715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.975680    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:20:05.978993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.978958    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:05.979093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.979029    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:05.979093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:05.979061    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:06.888157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:06.888109    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerStarted","Data":"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"}
Nov 02 20:20:06.888773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:06.888734    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerStarted","Data":"d29710e6f63e73befb580df438a5bfa2f4754ea1c8e35b06c6d001a38fe75037"}
Nov 02 20:20:06.971477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:06.971442    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:06.972100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:06.971831    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:06.974109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:06.973777    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:07.108277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:07.108217    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podStartSLOduration=3.108195066 podStartE2EDuration="3.108195066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:07.072387562 +0000 UTC m=+1695.946658477" watchObservedRunningTime="2026-11-02 20:20:07.108195066 +0000 UTC m=+1695.982465956"
Nov 02 20:20:07.336871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:07.331558    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:07.343237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:07.341152    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:07.896130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:07.896098    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:07.897015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:07.896997    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:20:09.532379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:09.532322    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" podUID="30d12fc0-613c-401a-8ba2-48df651f627e" containerName="payload-pre-processing" containerID="cri-o://085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3" gracePeriod=25
Nov 02 20:20:09.948679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:09.948651    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:20:10.049995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.047185    5099 generic.go:356] "Generic (PLEG): container finished" podID="30d12fc0-613c-401a-8ba2-48df651f627e" containerID="085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3" exitCode=0
Nov 02 20:20:10.049995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.049704    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"
Nov 02 20:20:10.050616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.050325    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" event={"ID":"30d12fc0-613c-401a-8ba2-48df651f627e","Type":"ContainerDied","Data":"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"}
Nov 02 20:20:10.050616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.050382    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8" event={"ID":"30d12fc0-613c-401a-8ba2-48df651f627e","Type":"ContainerDied","Data":"580e3872bcfce7ecd00d2e37801431ab249c2c0c67cec25745d5ba0eee6d14d0"}
Nov 02 20:20:10.050616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.050410    5099 scope.go:122] "RemoveContainer" containerID="085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"
Nov 02 20:20:10.114232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.114212    5099 scope.go:122] "RemoveContainer" containerID="085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"
Nov 02 20:20:10.114592 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:10.114538    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3\": container with ID starting with 085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3 not found: ID does not exist" containerID="085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"
Nov 02 20:20:10.114592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.114581    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3"} err="failed to get container status \"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3\": rpc error: code = NotFound desc = could not find container \"085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3\": container with ID starting with 085705a5be0604ac451b397f4981806528818553282ee786d8088382c5c38ad3 not found: ID does not exist"
Nov 02 20:20:10.119883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.119854    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\" (UniqueName: \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\") pod \"30d12fc0-613c-401a-8ba2-48df651f627e\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") "
Nov 02 20:20:10.120031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.120015    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\") pod \"30d12fc0-613c-401a-8ba2-48df651f627e\" (UID: \"30d12fc0-613c-401a-8ba2-48df651f627e\") "
Nov 02 20:20:10.122314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.122288    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume" pod "30d12fc0-613c-401a-8ba2-48df651f627e" (UID: "30d12fc0-613c-401a-8ba2-48df651f627e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:10.124520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.124484    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849" pod "30d12fc0-613c-401a-8ba2-48df651f627e" (UID: "30d12fc0-613c-401a-8ba2-48df651f627e"). InnerVolumeSpecName "kube-api-access-tc849". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:10.222124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.222088    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc849\" (UniqueName: \"kubernetes.io/projected/30d12fc0-613c-401a-8ba2-48df651f627e-kube-api-access-tc849\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:10.222124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.222124    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d12fc0-613c-401a-8ba2-48df651f627e-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:10.470060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.470021    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:20:10.517547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:10.517511    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55e81986-75bb5f7555-mgtm8"]
Nov 02 20:20:11.437974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:11.437937    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d12fc0-613c-401a-8ba2-48df651f627e" path="/var/lib/kubelet/pods/30d12fc0-613c-401a-8ba2-48df651f627e/volumes"
Nov 02 20:20:14.133763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.133726    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"]
Nov 02 20:20:14.183364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.183339    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f75722a-zx5hj"]
Nov 02 20:20:14.867531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.857128    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:20:14.878145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.878109    5099 cpu_manager.go:414] "containerMap: removing container" podUID="30d12fc0-613c-401a-8ba2-48df651f627e" containerName="payload-pre-processing"
Nov 02 20:20:14.878361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.878344    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d12fc0-613c-401a-8ba2-48df651f627e" containerName="payload-pre-processing"
Nov 02 20:20:14.878663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.878646    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d12fc0-613c-401a-8ba2-48df651f627e" containerName="payload-pre-processing"
Nov 02 20:20:14.887668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.887637    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:20:14.888057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.887986    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:14.942796 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.942748    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a51b3f\""
Nov 02 20:20:14.997488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:14.997433    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a51b3f-dockercfg-7vdhb\""
Nov 02 20:20:15.263866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.208866    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.263866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.208976    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcds\" (UniqueName: \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.276827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.212898    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:20:15.350957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.321451    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.350957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.321596    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcds\" (UniqueName: \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.351843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.351756    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.443804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.443746    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcds\" (UniqueName: \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\") pod \"payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.448172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.448138    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.459250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.459220    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e333a509-b023-4f15-8bbc-ab43fc2daaa6" path="/var/lib/kubelet/pods/e333a509-b023-4f15-8bbc-ab43fc2daaa6/volumes"
Nov 02 20:20:15.459814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.459772    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:20:15.534671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.534582    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.534963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.534930    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpfk\" (UniqueName: \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.549656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.549634    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:15.637460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.637248    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpfk\" (UniqueName: \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.638567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.637711    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.638693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.638673    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.693746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.693656    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpfk\" (UniqueName: \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\") pod \"payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.765865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.765827    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:15.932505 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:15.932464    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250118d2_58de_4f36_82b4_85528f5e6d1c.slice/crio-f12ee72922306292912aef62906addbe03ad6f0af47ab44d746ffaef87fb911d WatchSource:0}: Error finding container f12ee72922306292912aef62906addbe03ad6f0af47ab44d746ffaef87fb911d: Status 404 returned error can't find the container with id f12ee72922306292912aef62906addbe03ad6f0af47ab44d746ffaef87fb911d
Nov 02 20:20:15.934112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:15.934077    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:20:16.096139 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:16.096046    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:20:16.141563 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:16.141526    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca444a5_093b_4737_8949_7ec6a41941b7.slice/crio-40791847bc5230766ed5fdebbbc1a87dc3292d1d78ced238481e70e1fcf9aaea WatchSource:0}: Error finding container 40791847bc5230766ed5fdebbbc1a87dc3292d1d78ced238481e70e1fcf9aaea: Status 404 returned error can't find the container with id 40791847bc5230766ed5fdebbbc1a87dc3292d1d78ced238481e70e1fcf9aaea
Nov 02 20:20:16.142290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.142269    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:20:16.304666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.304621    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" event={"ID":"250118d2-58de-4f36-82b4-85528f5e6d1c","Type":"ContainerStarted","Data":"8ee90f21697dfde0e5e2d9b6150e89aad1f3e8980bd96aad94b3776e781c0e29"}
Nov 02 20:20:16.305190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.304691    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:16.305190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.304712    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" event={"ID":"250118d2-58de-4f36-82b4-85528f5e6d1c","Type":"ContainerStarted","Data":"f12ee72922306292912aef62906addbe03ad6f0af47ab44d746ffaef87fb911d"}
Nov 02 20:20:16.306647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.306618    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" event={"ID":"eca444a5-093b-4737-8949-7ec6a41941b7","Type":"ContainerStarted","Data":"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"}
Nov 02 20:20:16.306746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.306657    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" event={"ID":"eca444a5-093b-4737-8949-7ec6a41941b7","Type":"ContainerStarted","Data":"40791847bc5230766ed5fdebbbc1a87dc3292d1d78ced238481e70e1fcf9aaea"}
Nov 02 20:20:16.306746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.306690    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:16.415265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.415214    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" podStartSLOduration=2.41519991 podStartE2EDuration="2.41519991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:16.411581337 +0000 UTC m=+1705.285852226" watchObservedRunningTime="2026-11-02 20:20:16.41519991 +0000 UTC m=+1705.289470798"
Nov 02 20:20:16.519329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:16.519264    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" podStartSLOduration=2.519249336 podStartE2EDuration="2.519249336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:16.518022217 +0000 UTC m=+1705.392293105" watchObservedRunningTime="2026-11-02 20:20:16.519249336 +0000 UTC m=+1705.393520223"
Nov 02 20:20:20.951830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:20.951778    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:20:20.952247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:20.952077    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy" containerID="cri-o://926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d" gracePeriod=30
Nov 02 20:20:26.284922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.284886    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:20:26.393708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.393675    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.393891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.393734    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.393891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.393770    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.393891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.393855    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.394045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.393968    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.394045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394000    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:26.394154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394039    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.394154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394078    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.394264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394178    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:26.394319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394264    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.394372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394318    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:26.394444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394366    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:26.394506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394487    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:26.394554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.394294    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\") pod \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\" (UID: \"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f\") "
Nov 02 20:20:26.395184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.395161    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.395184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.395185    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.395373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.395194    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.395373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.395203    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.395373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.395212    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.396211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.396174    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:26.396316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.396247    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:26.396578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.396557    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:26.396673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.396590    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck" pod "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" (UID: "a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f"). InnerVolumeSpecName "kube-api-access-vwvck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:26.471804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.471607    5099 generic.go:356] "Generic (PLEG): container finished" podID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerID="926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d" exitCode=0
Nov 02 20:20:26.471804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.471680    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"
Nov 02 20:20:26.472069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.471922    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerDied","Data":"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"}
Nov 02 20:20:26.472069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.471970    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d" event={"ID":"a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f","Type":"ContainerDied","Data":"fca632f0231938e915e6e1f068a9ceb7a4a2b5bb5998a83b8cd6901285b43293"}
Nov 02 20:20:26.472069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.471997    5099 scope.go:122] "RemoveContainer" containerID="926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"
Nov 02 20:20:26.495226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.495193    5099 scope.go:122] "RemoveContainer" containerID="8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"
Nov 02 20:20:26.496763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.496734    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.496882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.496771    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.496882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.496802    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.496882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.496817    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvck\" (UniqueName: \"kubernetes.io/projected/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f-kube-api-access-vwvck\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:26.508014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.507985    5099 scope.go:122] "RemoveContainer" containerID="926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"
Nov 02 20:20:26.508408 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:26.508377    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d\": container with ID starting with 926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d not found: ID does not exist" containerID="926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"
Nov 02 20:20:26.508528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.508422    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d"} err="failed to get container status \"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d\": rpc error: code = NotFound desc = could not find container \"926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d\": container with ID starting with 926c54c1b082b7cbac4fb0d81b82726277ca8751069d46ce7bb45ee08d23a42d not found: ID does not exist"
Nov 02 20:20:26.508528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.508447    5099 scope.go:122] "RemoveContainer" containerID="8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"
Nov 02 20:20:26.508756 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:26.508730    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40\": container with ID starting with 8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40 not found: ID does not exist" containerID="8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"
Nov 02 20:20:26.508845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.508766    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40"} err="failed to get container status \"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40\": rpc error: code = NotFound desc = could not find container \"8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40\": container with ID starting with 8f7de75b7d48463b0bdc1fe45fd0df0e5a3ba042f1ba8c17aaea4aa6da1ddb40 not found: ID does not exist"
Nov 02 20:20:26.631472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.631420    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:20:26.635846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:26.635813    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7f75722a-openshift-default-ff7b75745-kx89d"]
Nov 02 20:20:27.264770 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:27.264730    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" path="/var/lib/kubelet/pods/a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f/volumes"
Nov 02 20:20:27.365968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:27.351991    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:20:27.365968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:27.353466    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:20:29.160805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:29.155453    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"]
Nov 02 20:20:29.196803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:29.193011    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-13c1d955-dh25t"]
Nov 02 20:20:29.322850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:29.322624    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b688f777-3b77-4d12-8dd7-534ea472cece" path="/var/lib/kubelet/pods/b688f777-3b77-4d12-8dd7-534ea472cece/volumes"
Nov 02 20:20:31.973189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.973144    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:20:31.974138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974115    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.974138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974136    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.974307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974155    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.974307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974160    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.974307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974289    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.974307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.974299    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dbe970-e0e1-42ba-b4c4-2ebb5d22528f" containerName="istio-proxy"
Nov 02 20:20:31.980110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:31.980080    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.072150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.072120    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:20:32.087718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.087684    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4dc8ca-openshift-default-dockercfg-dfbwg\""
Nov 02 20:20:32.091127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091090    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfmq\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.091351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091335    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.091461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091446    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.091578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091565    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.091709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091693    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.092381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.091908    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.092381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.092308    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.092381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.092366    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.092654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.092428    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206502    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfmq\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206562    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206592    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206665    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206739    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206859    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206928    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.206984    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.207059    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.216825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.207827    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.242932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.240185    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.242932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.240337    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.242932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.240851    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.358528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.347876    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.368653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.368611    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.413545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.411378    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfmq\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.445224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.445183    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.480848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.480709    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\") pod \"e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") " pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:32.596271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:32.596174    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:33.030349 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:33.030312    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19e0fd2_2a86_423a_be45_aaf6c12d4e2a.slice/crio-db6b275316faa72364e323def1ac5d4fb3c61c17dea2e51e96f558f280aa38a5 WatchSource:0}: Error finding container db6b275316faa72364e323def1ac5d4fb3c61c17dea2e51e96f558f280aa38a5: Status 404 returned error can't find the container with id db6b275316faa72364e323def1ac5d4fb3c61c17dea2e51e96f558f280aa38a5
Nov 02 20:20:33.031134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.031106    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:20:33.033390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.033357    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:33.033491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.033423    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:33.033491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.033451    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:33.044558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.044521    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"]
Nov 02 20:20:33.080507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.080450    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba4a9a3-2b4p8"]
Nov 02 20:20:33.260480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.260445    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c811d7d-97c0-45ef-aa4c-2d47f10e72d9" path="/var/lib/kubelet/pods/2c811d7d-97c0-45ef-aa4c-2d47f10e72d9/volumes"
Nov 02 20:20:33.388281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.388232    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerStarted","Data":"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"}
Nov 02 20:20:33.388491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.388299    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerStarted","Data":"db6b275316faa72364e323def1ac5d4fb3c61c17dea2e51e96f558f280aa38a5"}
Nov 02 20:20:33.501335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.501268    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podStartSLOduration=2.501255819 podStartE2EDuration="2.501255819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:33.498536836 +0000 UTC m=+1722.372807753" watchObservedRunningTime="2026-11-02 20:20:33.501255819 +0000 UTC m=+1722.375526771"
Nov 02 20:20:33.596488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:33.596453    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:34.602863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:34.602834    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:35.425397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.425356    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:35.426852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.426824    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:20:35.444762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.444729    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:20:35.445037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.445007    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy" containerID="cri-o://68a14f04907ad7c9c8b22398b4057c5b0840cfd9bef4e4d2f1073a4ba42f3b2b" gracePeriod=30
Nov 02 20:20:35.488431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.488402    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:20:35.488622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.488600    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" containerID="cri-o://657550c75d17009eb5308a9d50686b2cc1dc771205a46b726ac80061a89f297b" gracePeriod=30
Nov 02 20:20:35.492886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.492850    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:20:35.493078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.493035    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" containerID="cri-o://3dd834a808740464c1cbaec0873a4f1911f4d4dbee4a88a0269c01521d9d7bf4" gracePeriod=30
Nov 02 20:20:35.536048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.535986    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:20:35.536266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:35.536238    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy" containerID="cri-o://efd562d43958ae12ee0a6e77458b61b9328efa008273a040a9db372453843d2c" gracePeriod=30
Nov 02 20:20:36.535841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.526910    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:20:36.535841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.527150    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:20:36.535841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.527410    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy" containerID="cri-o://4673aa0e92cee412664753b60969ae36515a80054813e33fa2ce34849fc6c09a" gracePeriod=30
Nov 02 20:20:36.535841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.527163    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy" containerID="cri-o://f19b6447e0a4e5a05b7263063015ea8c9b82f386e4b5a313c0144e806aa587d9" gracePeriod=30
Nov 02 20:20:36.537515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.536998    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:20:36.537515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.537220    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" containerID="cri-o://5d38b682a7008c1c841f8314a0f3038bcb08e6a7264d1027916715784adb25fa" gracePeriod=30
Nov 02 20:20:36.546125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.546102    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:20:36.546476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:36.546454    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" containerID="cri-o://4f260fa5b99e377caf79cae122be6d1036e43f3e3ff8310ed5b20856974a93eb" gracePeriod=30
Nov 02 20:20:38.091392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:38.091351    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:20:38.091937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:38.091409    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:20:39.432042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:39.432004    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:20:39.433588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:39.433555    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:20:40.121842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.117823    5099 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:20:40.121842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.117898    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:20:40.255808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.255266    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"]
Nov 02 20:20:40.289835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.289797    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55e81986-ckrwm"]
Nov 02 20:20:40.346808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.341103    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:20:40.378868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.377097    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.382808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.381795    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:20:40.450278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.440257    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4dc8ca\""
Nov 02 20:20:40.450278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.440505    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4dc8ca-dockercfg-65xjc\""
Nov 02 20:20:40.604309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.568799    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.654850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.605024    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnkf\" (UniqueName: \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.712596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.694524    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:20:40.716929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.716762    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.717439 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.717406    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnkf\" (UniqueName: \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.749805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.730409    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.810807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.810249    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:40.890225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.886864    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:20:40.903125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.903020    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnkf\" (UniqueName: \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\") pod \"payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:40.913890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.913799    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:20:40.914344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.914291    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy" containerID="cri-o://39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9" gracePeriod=30
Nov 02 20:20:40.936328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.927514    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmwt\" (UniqueName: \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:40.936328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:40.927592    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.042242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.038478    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmwt\" (UniqueName: \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.042242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.038571    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.066853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.046416    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:41.204303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.077118    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.374992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.373863    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmwt\" (UniqueName: \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\") pod \"payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.510924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.505134    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d39233-093c-454e-8952-8c81f1839c32" path="/var/lib/kubelet/pods/85d39233-093c-454e-8952-8c81f1839c32/volumes"
Nov 02 20:20:41.617852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.596454    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:41.904040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.903994    5099 generic.go:356] "Generic (PLEG): container finished" podID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerID="f19b6447e0a4e5a05b7263063015ea8c9b82f386e4b5a313c0144e806aa587d9" exitCode=0
Nov 02 20:20:41.904505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.904317    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" event={"ID":"3b23f814-fed9-40fa-8551-ce1f6715ffc8","Type":"ContainerDied","Data":"f19b6447e0a4e5a05b7263063015ea8c9b82f386e4b5a313c0144e806aa587d9"}
Nov 02 20:20:41.907328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.906695    5099 generic.go:356] "Generic (PLEG): container finished" podID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerID="657550c75d17009eb5308a9d50686b2cc1dc771205a46b726ac80061a89f297b" exitCode=0
Nov 02 20:20:41.907328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.906764    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" event={"ID":"700f978f-63c3-4947-b7b6-c925cea7d1fb","Type":"ContainerDied","Data":"657550c75d17009eb5308a9d50686b2cc1dc771205a46b726ac80061a89f297b"}
Nov 02 20:20:41.907328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.906816    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q" event={"ID":"700f978f-63c3-4947-b7b6-c925cea7d1fb","Type":"ContainerDied","Data":"7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48"}
Nov 02 20:20:41.907328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.906834    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0aea9dad3fad04d38ef808cc7be79547fe1ab4bd2dc51e0890ebacad633e48"
Nov 02 20:20:41.909165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.909024    5099 generic.go:356] "Generic (PLEG): container finished" podID="39a701bd-7a2c-4016-a17c-945e6208040f" containerID="4f260fa5b99e377caf79cae122be6d1036e43f3e3ff8310ed5b20856974a93eb" exitCode=0
Nov 02 20:20:41.909165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.909097    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" event={"ID":"39a701bd-7a2c-4016-a17c-945e6208040f","Type":"ContainerDied","Data":"4f260fa5b99e377caf79cae122be6d1036e43f3e3ff8310ed5b20856974a93eb"}
Nov 02 20:20:41.912683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.912298    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:20:41.914339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.914316    5099 generic.go:356] "Generic (PLEG): container finished" podID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerID="efd562d43958ae12ee0a6e77458b61b9328efa008273a040a9db372453843d2c" exitCode=0
Nov 02 20:20:41.914694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.914678    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" event={"ID":"20521a86-d32b-40cf-b237-c4d7e6b4e087","Type":"ContainerDied","Data":"efd562d43958ae12ee0a6e77458b61b9328efa008273a040a9db372453843d2c"}
Nov 02 20:20:41.915680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.915655    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.915947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.915929    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.916065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.916052    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.919277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919257    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.919652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919612    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:41.919765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919683    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.919863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919770    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.919863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919824    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.919980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919931    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.920039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.919984    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\") pod \"700f978f-63c3-4947-b7b6-c925cea7d1fb\" (UID: \"700f978f-63c3-4947-b7b6-c925cea7d1fb\") "
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.920455    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.920672    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.920892    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921279    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921315    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921329    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921343    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921445    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:41.921640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.921619    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:41.922693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.922659    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:41.924015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.923940    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "kube-api-access-w92vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:41.925649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.924573    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token" pod "700f978f-63c3-4947-b7b6-c925cea7d1fb" (UID: "700f978f-63c3-4947-b7b6-c925cea7d1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:41.952576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.951269    5099 generic.go:356] "Generic (PLEG): container finished" podID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerID="4673aa0e92cee412664753b60969ae36515a80054813e33fa2ce34849fc6c09a" exitCode=0
Nov 02 20:20:41.952576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.951400    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" event={"ID":"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad","Type":"ContainerDied","Data":"4673aa0e92cee412664753b60969ae36515a80054813e33fa2ce34849fc6c09a"}
Nov 02 20:20:41.967139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.966939    5099 generic.go:356] "Generic (PLEG): container finished" podID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerID="5d38b682a7008c1c841f8314a0f3038bcb08e6a7264d1027916715784adb25fa" exitCode=0
Nov 02 20:20:41.967139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.967067    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" event={"ID":"9f327d56-dbdf-49fd-a8ec-d29189565296","Type":"ContainerDied","Data":"5d38b682a7008c1c841f8314a0f3038bcb08e6a7264d1027916715784adb25fa"}
Nov 02 20:20:41.988808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.981158    5099 generic.go:356] "Generic (PLEG): container finished" podID="1c2f4829-1312-4341-aed0-9bd94add9887" containerID="3dd834a808740464c1cbaec0873a4f1911f4d4dbee4a88a0269c01521d9d7bf4" exitCode=0
Nov 02 20:20:41.988808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:41.981456    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" event={"ID":"1c2f4829-1312-4341-aed0-9bd94add9887","Type":"ContainerDied","Data":"3dd834a808740464c1cbaec0873a4f1911f4d4dbee4a88a0269c01521d9d7bf4"}
Nov 02 20:20:42.011870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.011681    5099 generic.go:356] "Generic (PLEG): container finished" podID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerID="68a14f04907ad7c9c8b22398b4057c5b0840cfd9bef4e4d2f1073a4ba42f3b2b" exitCode=0
Nov 02 20:20:42.011870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.011768    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" event={"ID":"ff507b91-f192-44ce-b32a-287ba00cc9c6","Type":"ContainerDied","Data":"68a14f04907ad7c9c8b22398b4057c5b0840cfd9bef4e4d2f1073a4ba42f3b2b"}
Nov 02 20:20:42.038082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.038048    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700f978f-63c3-4947-b7b6-c925cea7d1fb-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.038572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.038550    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92vd\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-kube-api-access-w92vd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.038739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.038724    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.044813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.038844    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.045352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.045328    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700f978f-63c3-4947-b7b6-c925cea7d1fb-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.267312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.266936    5099 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 02 20:20:42.267312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.266996    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 02 20:20:42.317860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.302004    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:20:42.335891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.335094    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:20:42.401475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.401436    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.401762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.401739    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.401991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.401970    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.402172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.402153    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.402345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.402329    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.408752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.402505    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.428761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.428736    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.428964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.428943    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.429139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.429122    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.429259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.429245    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.429499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.429388    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.429620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.429595    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.430808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.429882    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.573908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.573830    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.575104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.574441    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.575313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.575291    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.575510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.575492    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\") pod \"20521a86-d32b-40cf-b237-c4d7e6b4e087\" (UID: \"20521a86-d32b-40cf-b237-c4d7e6b4e087\") "
Nov 02 20:20:42.575676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.575660    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\") pod \"1c2f4829-1312-4341-aed0-9bd94add9887\" (UID: \"1c2f4829-1312-4341-aed0-9bd94add9887\") "
Nov 02 20:20:42.578553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.428006    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:20:42.578764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.424799    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.578934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.425021    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.579020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.515529    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.579099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.537952    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.579191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.568835    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:42.579270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.569346    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.579340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.569416    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.580515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.580156    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.580515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.580351    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:42.582104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.582081    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.618513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.618469    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.622210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.622156    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "kube-api-access-h2z6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.624193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.624166    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.626151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.626110    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:42.626310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.626273    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:42.636702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.636664    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.657604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.639932    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token" pod "20521a86-d32b-40cf-b237-c4d7e6b4e087" (UID: "20521a86-d32b-40cf-b237-c4d7e6b4e087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.657604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.643658    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb" pod "1c2f4829-1312-4341-aed0-9bd94add9887" (UID: "1c2f4829-1312-4341-aed0-9bd94add9887"). InnerVolumeSpecName "kube-api-access-7zmkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.682863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.682708    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683114    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683464    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683619    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683746    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.683924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.683907    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.684179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.684160    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.684370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.684355    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\") pod \"ff507b91-f192-44ce-b32a-287ba00cc9c6\" (UID: \"ff507b91-f192-44ce-b32a-287ba00cc9c6\") "
Nov 02 20:20:42.685281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685255    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685421    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685527    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685617    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685703    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20521a86-d32b-40cf-b237-c4d7e6b4e087-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685813    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.685926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.685914    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686001    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2z6j\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-kube-api-access-h2z6j\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686088    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f4829-1312-4341-aed0-9bd94add9887-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686170    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686249    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmkb\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-kube-api-access-7zmkb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686336    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686439 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686429    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c2f4829-1312-4341-aed0-9bd94add9887-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686504    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686595    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686677    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c2f4829-1312-4341-aed0-9bd94add9887-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686761    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20521a86-d32b-40cf-b237-c4d7e6b4e087-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.686904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.686890    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c2f4829-1312-4341-aed0-9bd94add9887-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.687229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.687207    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.687694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.687673    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:42.694537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.694194    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.703584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.699931    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.709817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.708863    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.710283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.710252    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.710911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.710877    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "kube-api-access-5fzj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:42.714284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.714214    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:42.720763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.720737    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy" pod "ff507b91-f192-44ce-b32a-287ba00cc9c6" (UID: "ff507b91-f192-44ce-b32a-287ba00cc9c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.750105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.749979    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802316    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802386    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802403    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802418    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802434    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzj4\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-kube-api-access-5fzj4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802449    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802463    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff507b91-f192-44ce-b32a-287ba00cc9c6-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802479    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.802815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.802493    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff507b91-f192-44ce-b32a-287ba00cc9c6-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:42.891984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.891952    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:20:42.897610 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.897581    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:20:42.900554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.900451    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:20:42.905769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.905732    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.906088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.906063    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.906362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.906336    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.906581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.906557    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.906763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.906749    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.906970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.906906    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.907184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.907165    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.907412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.907396    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.907569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.907553    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\") pod \"39a701bd-7a2c-4016-a17c-945e6208040f\" (UID: \"39a701bd-7a2c-4016-a17c-945e6208040f\") "
Nov 02 20:20:42.910190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.910162    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.910479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.910451    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.911854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.911823    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.912717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.912682    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:42.913486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:42.913466    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:43.066941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.066683    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.066941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.066776    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:43.066941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.066852    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.066941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.066914    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn" pod "39a701bd-7a2c-4016-a17c-945e6208040f" (UID: "39a701bd-7a2c-4016-a17c-945e6208040f"). InnerVolumeSpecName "kube-api-access-2qbhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.068386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.068331    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.068641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.068621    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.068852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.068837    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.068984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.068955    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.069095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069083    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.069213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069201    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.069321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069308    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.069429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069417    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.069533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069521    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.069643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069632    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.069773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.069761    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.070069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070053    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.070183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070170    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.070300 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070288    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.070395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070383    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.070509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070497    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.070605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070592    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.070745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070732    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.070932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.070914    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.071111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.071098    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.071223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.071211    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.071339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.071326    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.071449 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.071436    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.071892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.071532    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.072116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.072090    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\") pod \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\" (UID: \"3b23f814-fed9-40fa-8551-ce1f6715ffc8\") "
Nov 02 20:20:43.072267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.072249    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\") pod \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\" (UID: \"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad\") "
Nov 02 20:20:43.072375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.072362    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\") pod \"9f327d56-dbdf-49fd-a8ec-d29189565296\" (UID: \"9f327d56-dbdf-49fd-a8ec-d29189565296\") "
Nov 02 20:20:43.073307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.073280    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a701bd-7a2c-4016-a17c-945e6208040f-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.075734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.075710    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.075882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.075866    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.075969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.075959    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.076052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076042    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a701bd-7a2c-4016-a17c-945e6208040f-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.076129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076119    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.076203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076193    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.076282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076271    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qbhn\" (UniqueName: \"kubernetes.io/projected/39a701bd-7a2c-4016-a17c-945e6208040f-kube-api-access-2qbhn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.076362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076351    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a701bd-7a2c-4016-a17c-945e6208040f-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076912    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.076977    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.077181    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.077263    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.077394    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.077585    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.077972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.077886    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.082509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.082452    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.115850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.099147    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:20:43.117360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.109997    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.117502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.110044    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.117592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.110828    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:43.117667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.111105    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.117738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.111385    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:43.117866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.111622    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.127521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.126085    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.127521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.126941    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.127521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.127006    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:43.127521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.127101    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.127521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.127165    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "kube-api-access-tzlt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.129718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.129622    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo" pod "3b23f814-fed9-40fa-8551-ce1f6715ffc8" (UID: "3b23f814-fed9-40fa-8551-ce1f6715ffc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:43.131829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.131307    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "kube-api-access-j5tnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.131829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.131442    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.131829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.131770    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.132059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.131966    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:43.132476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.132445    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "kube-api-access-9mfb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.132665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.132641    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy" pod "9f327d56-dbdf-49fd-a8ec-d29189565296" (UID: "9f327d56-dbdf-49fd-a8ec-d29189565296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:43.132838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.132776    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token" pod "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" (UID: "e4906bc4-39b8-4d6e-ac58-a2c52fd55bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205230    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mfb4\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-kube-api-access-9mfb4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205270    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205288    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205312    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205326    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205339    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205352    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205365    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205379    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205392    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205403    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205416    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205429    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205443    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205456    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b23f814-fed9-40fa-8551-ce1f6715ffc8-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205469    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205483    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205498    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205512    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205526    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205539    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205554    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205569    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f327d56-dbdf-49fd-a8ec-d29189565296-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205585    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205600    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzlt6\" (UniqueName: \"kubernetes.io/projected/3b23f814-fed9-40fa-8551-ce1f6715ffc8-kube-api-access-tzlt6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205615    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.206967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.205628    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5tnq\" (UniqueName: \"kubernetes.io/projected/9f327d56-dbdf-49fd-a8ec-d29189565296-kube-api-access-j5tnq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:43.211164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.210178    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98" event={"ID":"ff507b91-f192-44ce-b32a-287ba00cc9c6","Type":"ContainerDied","Data":"518b787f2073892adf05d7f7a89d11850cf6f91c5850d6c85720bbc7dacc7976"}
Nov 02 20:20:43.211164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.210245    5099 scope.go:122] "RemoveContainer" containerID="68a14f04907ad7c9c8b22398b4057c5b0840cfd9bef4e4d2f1073a4ba42f3b2b"
Nov 02 20:20:43.211164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.210419    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"
Nov 02 20:20:43.215892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.215356    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"
Nov 02 20:20:43.215892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.215456    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g" event={"ID":"3b23f814-fed9-40fa-8551-ce1f6715ffc8","Type":"ContainerDied","Data":"383b8d25dbd151640e9d4f710164485c7b82b3da4a9f00add0b9852d72bd10e8"}
Nov 02 20:20:43.219589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.219559    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk" event={"ID":"39a701bd-7a2c-4016-a17c-945e6208040f","Type":"ContainerDied","Data":"6dc576a42a08d65b6f276338f533334935ffd844877662a958f4c63b2e0afa96"}
Nov 02 20:20:43.219694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.219674    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"
Nov 02 20:20:43.229609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.222367    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb" event={"ID":"20521a86-d32b-40cf-b237-c4d7e6b4e087","Type":"ContainerDied","Data":"5f837b9914660c1075ab8e1fc5ec136b946f848616e0565d9655657c6e1d0956"}
Nov 02 20:20:43.229609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.222462    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"
Nov 02 20:20:43.229814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.229640    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4" event={"ID":"e4906bc4-39b8-4d6e-ac58-a2c52fd55bad","Type":"ContainerDied","Data":"0ba881965d0a54b50e5098fd6b04173a9ec05fb6b9d59207cbdb2a991090bee5"}
Nov 02 20:20:43.229814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.229728    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"
Nov 02 20:20:43.235440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.232854    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm" event={"ID":"9f327d56-dbdf-49fd-a8ec-d29189565296","Type":"ContainerDied","Data":"0f140b2a33c6104d82a9670b1c32cb2043aa3262bc5eeae8a78245f4ebf646c0"}
Nov 02 20:20:43.235440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.232943    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"
Nov 02 20:20:43.283426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.283392    5099 scope.go:122] "RemoveContainer" containerID="f19b6447e0a4e5a05b7263063015ea8c9b82f386e4b5a313c0144e806aa587d9"
Nov 02 20:20:43.286574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.284928    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"
Nov 02 20:20:43.289430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.289408    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"
Nov 02 20:20:43.313447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.313325    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p" event={"ID":"1c2f4829-1312-4341-aed0-9bd94add9887","Type":"ContainerDied","Data":"2ad9d52cbab1913238d17877c0f2e1aaed1d194c961d13db0eb94175e48ec92a"}
Nov 02 20:20:43.384446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.384421    5099 scope.go:122] "RemoveContainer" containerID="4f260fa5b99e377caf79cae122be6d1036e43f3e3ff8310ed5b20856974a93eb"
Nov 02 20:20:43.430522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.430501    5099 scope.go:122] "RemoveContainer" containerID="efd562d43958ae12ee0a6e77458b61b9328efa008273a040a9db372453843d2c"
Nov 02 20:20:43.450196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.450170    5099 scope.go:122] "RemoveContainer" containerID="4673aa0e92cee412664753b60969ae36515a80054813e33fa2ce34849fc6c09a"
Nov 02 20:20:43.462887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.462856    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:20:43.555359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.555330    5099 scope.go:122] "RemoveContainer" containerID="5d38b682a7008c1c841f8314a0f3038bcb08e6a7264d1027916715784adb25fa"
Nov 02 20:20:43.578553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.578404    5099 scope.go:122] "RemoveContainer" containerID="3dd834a808740464c1cbaec0873a4f1911f4d4dbee4a88a0269c01521d9d7bf4"
Nov 02 20:20:43.582402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.582383    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:20:43.590555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.590523    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-wvsjk"]
Nov 02 20:20:43.686197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.686166    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:20:43.690229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.690203    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-q6m98"]
Nov 02 20:20:43.840959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.840914    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:20:43.845013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.844984    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-v4t6g"]
Nov 02 20:20:43.997933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:43.997847    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:20:44.002107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.002075    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-dccqb"]
Nov 02 20:20:44.158047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.158010    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:20:44.164357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.164322    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-kp58p"]
Nov 02 20:20:44.289745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.289665    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:20:44.326748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.320397    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-kw2v4"]
Nov 02 20:20:44.374225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.373664    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" event={"ID":"9b15509e-20b1-4d38-bf80-db208b0e3b90","Type":"ContainerStarted","Data":"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"}
Nov 02 20:20:44.374225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.373729    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" event={"ID":"9b15509e-20b1-4d38-bf80-db208b0e3b90","Type":"ContainerStarted","Data":"dc38bfb28282c4a00b25c38fa2d52ce3e881ca948cf9f71a4a42dd42dae80162"}
Nov 02 20:20:44.375087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.375062    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:20:44.380328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.380302    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" event={"ID":"46e56843-6e9d-43ad-b0bf-3c114523f415","Type":"ContainerStarted","Data":"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"}
Nov 02 20:20:44.380496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.380481    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" event={"ID":"46e56843-6e9d-43ad-b0bf-3c114523f415","Type":"ContainerStarted","Data":"5916d3f05b952f3c2347b180272c792b897820925a79633754f4b5753d4d3e1f"}
Nov 02 20:20:44.381423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.381404    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:44.394005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.393982    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:20:44.397492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.397469    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-4bf9q"]
Nov 02 20:20:44.501266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.501235    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:20:44.508213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.508184    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-nfslm"]
Nov 02 20:20:44.610550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.610500    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" podStartSLOduration=4.610484796 podStartE2EDuration="4.610484796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:44.610110377 +0000 UTC m=+1733.484381260" watchObservedRunningTime="2026-11-02 20:20:44.610484796 +0000 UTC m=+1733.484755684"
Nov 02 20:20:44.720261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:44.720197    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" podStartSLOduration=4.7201827640000005 podStartE2EDuration="4.720182764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:44.716873048 +0000 UTC m=+1733.591143934" watchObservedRunningTime="2026-11-02 20:20:44.720182764 +0000 UTC m=+1733.594453652"
Nov 02 20:20:45.278903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.278861    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" path="/var/lib/kubelet/pods/1c2f4829-1312-4341-aed0-9bd94add9887/volumes"
Nov 02 20:20:45.279718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.279697    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" path="/var/lib/kubelet/pods/20521a86-d32b-40cf-b237-c4d7e6b4e087/volumes"
Nov 02 20:20:45.280476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.280456    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" path="/var/lib/kubelet/pods/39a701bd-7a2c-4016-a17c-945e6208040f/volumes"
Nov 02 20:20:45.281686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.281665    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" path="/var/lib/kubelet/pods/3b23f814-fed9-40fa-8551-ce1f6715ffc8/volumes"
Nov 02 20:20:45.282409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.282387    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" path="/var/lib/kubelet/pods/700f978f-63c3-4947-b7b6-c925cea7d1fb/volumes"
Nov 02 20:20:45.283150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.283130    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" path="/var/lib/kubelet/pods/9f327d56-dbdf-49fd-a8ec-d29189565296/volumes"
Nov 02 20:20:45.284433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.284411    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" path="/var/lib/kubelet/pods/e4906bc4-39b8-4d6e-ac58-a2c52fd55bad/volumes"
Nov 02 20:20:45.285173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:45.285152    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" path="/var/lib/kubelet/pods/ff507b91-f192-44ce-b32a-287ba00cc9c6/volumes"
Nov 02 20:20:46.107521 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:46.107476    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:20:46.250005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.249976    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:46.389745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.389704    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.389935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.389773    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.389935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.389843    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.389935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.389899    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.390111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.389945    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.390111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390003    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.390212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390147    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.390212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390181    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390373    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\") pod \"df2b1b60-82f8-4641-8919-5eec77dae2c9\" (UID: \"df2b1b60-82f8-4641-8919-5eec77dae2c9\") "
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390406    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390757    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390806    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390838    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.391366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.390870    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.391793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.391467    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2b1b60-82f8-4641-8919-5eec77dae2c9-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.391793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.391491    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.391793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.391506    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.391793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.391520    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.391793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.391533    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.393212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.393180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:46.395862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.395777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:20:46.395978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.395913    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "kube-api-access-dxgm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:20:46.396898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.396860    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo" pod "df2b1b60-82f8-4641-8919-5eec77dae2c9" (UID: "df2b1b60-82f8-4641-8919-5eec77dae2c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:20:46.411145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.411117    5099 generic.go:356] "Generic (PLEG): container finished" podID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerID="39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9" exitCode=0
Nov 02 20:20:46.411262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.411173    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"
Nov 02 20:20:46.411262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.411196    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerDied","Data":"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"}
Nov 02 20:20:46.411262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.411257    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6" event={"ID":"df2b1b60-82f8-4641-8919-5eec77dae2c9","Type":"ContainerDied","Data":"e5987d30f1139999599bf39e1d09233afbe429abec5851e7c8c6e4960a9026ee"}
Nov 02 20:20:46.411431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.411287    5099 scope.go:122] "RemoveContainer" containerID="39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"
Nov 02 20:20:46.441912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.441884    5099 scope.go:122] "RemoveContainer" containerID="0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"
Nov 02 20:20:46.454627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.454597    5099 scope.go:122] "RemoveContainer" containerID="39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"
Nov 02 20:20:46.455106 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:46.454986    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9\": container with ID starting with 39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9 not found: ID does not exist" containerID="39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"
Nov 02 20:20:46.455106 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.455021    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9"} err="failed to get container status \"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9\": rpc error: code = NotFound desc = could not find container \"39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9\": container with ID starting with 39cf0dcdff95c3b655d4cabb84db1be0e304ddd245c0e0d5190c74776073cba9 not found: ID does not exist"
Nov 02 20:20:46.455106 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.455043    5099 scope.go:122] "RemoveContainer" containerID="0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"
Nov 02 20:20:46.455396 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:20:46.455375    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8\": container with ID starting with 0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8 not found: ID does not exist" containerID="0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"
Nov 02 20:20:46.455493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.455399    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8"} err="failed to get container status \"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8\": rpc error: code = NotFound desc = could not find container \"0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8\": container with ID starting with 0892986a0840e148ae51a1770331e01bf7276bfef3466e7d8125643d2ffac8a8 not found: ID does not exist"
Nov 02 20:20:46.493622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.493541    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.493622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.493571    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgm4\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-kube-api-access-dxgm4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.493622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.493580    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.493622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.493589    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2b1b60-82f8-4641-8919-5eec77dae2c9-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:20:46.574422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.574376    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:20:46.579357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:46.579320    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-55e81986-openshift-default-cbbfdb69f-4tqh6"]
Nov 02 20:20:47.263643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:47.263606    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" path="/var/lib/kubelet/pods/df2b1b60-82f8-4641-8919-5eec77dae2c9/volumes"
Nov 02 20:20:48.434546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.434514    5099 generic.go:356] "Generic (PLEG): container finished" podID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerID="fe92a33b313e7cfb7934e78996f4b8107d0f4a2c03f974e81a4d9bacf04fba9f" exitCode=0
Nov 02 20:20:48.435007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.434557    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerDied","Data":"fe92a33b313e7cfb7934e78996f4b8107d0f4a2c03f974e81a4d9bacf04fba9f"}
Nov 02 20:20:48.435052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.435005    5099 scope.go:122] "RemoveContainer" containerID="fe92a33b313e7cfb7934e78996f4b8107d0f4a2c03f974e81a4d9bacf04fba9f"
Nov 02 20:20:48.436665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.436633    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:48.436822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.436721    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:48.436822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:48.436767    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:20:49.474174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:49.474124    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerStarted","Data":"23f8a696de3e0b7b926d4956fc0249618c85a2c867abdb1dff04414e74537c5c"}
Nov 02 20:20:50.263411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.263376    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:20:50.263855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.263816    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 20:20:50.270104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.270071    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.944052    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945110    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945132    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945158    5099 cpu_manager.go:414] "containerMap: removing container" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945167    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945182    5099 cpu_manager.go:414] "containerMap: removing container" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945190    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945205    5099 cpu_manager.go:414] "containerMap: removing container" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945212    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945245    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945252    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945265    5099 cpu_manager.go:414] "containerMap: removing container" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945272    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945285    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945366    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945401    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945474    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945519    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945527    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945551    5099 cpu_manager.go:414] "containerMap: removing container" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945558    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945833    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a701bd-7a2c-4016-a17c-945e6208040f" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945855    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="700f978f-63c3-4947-b7b6-c925cea7d1fb" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945869    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2f4829-1312-4341-aed0-9bd94add9887" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945877    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945886    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f327d56-dbdf-49fd-a8ec-d29189565296" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945898    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff507b91-f192-44ce-b32a-287ba00cc9c6" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945910    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b23f814-fed9-40fa-8551-ce1f6715ffc8" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945923    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="20521a86-d32b-40cf-b237-c4d7e6b4e087" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945934    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4906bc4-39b8-4d6e-ac58-a2c52fd55bad" containerName="istio-proxy"
Nov 02 20:20:50.946805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:50.945945    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2b1b60-82f8-4641-8919-5eec77dae2c9" containerName="istio-proxy"
Nov 02 20:20:51.157806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.134078    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.256866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.230967    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5fc50d-dockercfg-w5gq8\""
Nov 02 20:20:51.270860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.270007    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5fc50d\""
Nov 02 20:20:51.274933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.274899    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 20:20:51.285071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.285024    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 20:20:51.367809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.360798    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:20:51.413805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.368945    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.413805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.369180    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6jz\" (UniqueName: \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.571944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.565321    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6jz\" (UniqueName: \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.573626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.572463    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.594390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.594361    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:20:51.614344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.613433    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:20:51.614344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.613568    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.621486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.621449    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6jz\" (UniqueName: \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.634961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.634930    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5fc50d\""
Nov 02 20:20:51.646819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.643875    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.684752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.684660    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:20:51.796988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.796951    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gh4h\" (UniqueName: \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.797163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.797068    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.898884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.898849    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gh4h\" (UniqueName: \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.899064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.898937    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.899669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.899651    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.913589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.913566    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5fc50d-dockercfg-w5gq8\""
Nov 02 20:20:51.915758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.915745    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:51.979457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.979422    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gh4h\" (UniqueName: \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\") pod \"payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:51.986465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:51.986284    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:52.270873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.270048    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 20:20:52.270873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.270107    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 20:20:52.330314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.330282    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:20:52.330484 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:52.330453    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429aa04f_9423_4497_ad76_10359fb9fbdd.slice/crio-70ac0c989dc8d1788f64d74f7abe7dda93093058047484fe3bcf6c265a606092 WatchSource:0}: Error finding container 70ac0c989dc8d1788f64d74f7abe7dda93093058047484fe3bcf6c265a606092: Status 404 returned error can't find the container with id 70ac0c989dc8d1788f64d74f7abe7dda93093058047484fe3bcf6c265a606092
Nov 02 20:20:52.426989 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:20:52.426961    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d2072d_35a9_4a61_8229_f3d167f3e0fd.slice/crio-ccbd92fef878adf94244b9879c683b31ee4d2935b8ed247a290fe4e26e6b911d WatchSource:0}: Error finding container ccbd92fef878adf94244b9879c683b31ee4d2935b8ed247a290fe4e26e6b911d: Status 404 returned error can't find the container with id ccbd92fef878adf94244b9879c683b31ee4d2935b8ed247a290fe4e26e6b911d
Nov 02 20:20:52.427154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.427126    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:20:52.671807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.669696    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" event={"ID":"429aa04f-9423-4497-ad76-10359fb9fbdd","Type":"ContainerStarted","Data":"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"}
Nov 02 20:20:52.671807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.669770    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" event={"ID":"429aa04f-9423-4497-ad76-10359fb9fbdd","Type":"ContainerStarted","Data":"70ac0c989dc8d1788f64d74f7abe7dda93093058047484fe3bcf6c265a606092"}
Nov 02 20:20:52.671807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.670972    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:20:52.675642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.675611    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" event={"ID":"26d2072d-35a9-4a61-8229-f3d167f3e0fd","Type":"ContainerStarted","Data":"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"}
Nov 02 20:20:52.675815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.675664    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" event={"ID":"26d2072d-35a9-4a61-8229-f3d167f3e0fd","Type":"ContainerStarted","Data":"ccbd92fef878adf94244b9879c683b31ee4d2935b8ed247a290fe4e26e6b911d"}
Nov 02 20:20:52.675815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.675792    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:20:52.780261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.780147    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" podStartSLOduration=2.780134341 podStartE2EDuration="2.780134341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:52.779089959 +0000 UTC m=+1741.653360849" watchObservedRunningTime="2026-11-02 20:20:52.780134341 +0000 UTC m=+1741.654405228"
Nov 02 20:20:52.795950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.795915    5099 scope.go:122] "RemoveContainer" containerID="317c244de885aa8dadb7039e575cedbff4a149689731c4b016730e8c7cfe832d"
Nov 02 20:20:52.807105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.807082    5099 scope.go:122] "RemoveContainer" containerID="347916da24bd1496bca3b9e5f6123cd1e616e78f0194709cb06eed203bf544fc"
Nov 02 20:20:52.816988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.816969    5099 scope.go:122] "RemoveContainer" containerID="61d15c6e8836eb4664d988a204aa1f92b2587933573445887506360523ea4aa8"
Nov 02 20:20:52.826577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.826549    5099 scope.go:122] "RemoveContainer" containerID="657550c75d17009eb5308a9d50686b2cc1dc771205a46b726ac80061a89f297b"
Nov 02 20:20:52.860079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.860047    5099 scope.go:122] "RemoveContainer" containerID="424d3692c8996e960800e3fb27c023eed834495cdcee3402d9574476db418759"
Nov 02 20:20:52.882992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.882689    5099 scope.go:122] "RemoveContainer" containerID="2eac2bf66c956a9e081dcb187952821f90b7e2dfe7c8f4ed89ca52fc362b35ab"
Nov 02 20:20:52.946303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:52.946255    5099 scope.go:122] "RemoveContainer" containerID="c1fde8b92950ab2dc04150d8d63140de10b67d9c0c12c78a38e682e8a7da4104"
Nov 02 20:20:53.001401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:53.001242    5099 scope.go:122] "RemoveContainer" containerID="b899914bc52bec6f8cf864140b607366e0823c8e7fb295927f9d82c13bf40110"
Nov 02 20:20:53.142047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:53.140965    5099 scope.go:122] "RemoveContainer" containerID="e357f403f410ad0bd4d3ba77254dcc10055976faaae0265a4d6ac5ff851f3fad"
Nov 02 20:20:53.231828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:53.226376    5099 scope.go:122] "RemoveContainer" containerID="ee86786f59ba66bdb490f1ed03336a38af9bef1d02db963e50a18829841c71ce"
Nov 02 20:20:53.274805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:53.273735    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 20:20:53.274805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:53.273810    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 20:20:54.259807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:54.259579    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 20:20:54.259807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:54.259641    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 20:20:55.296650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:55.296618    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:20:55.298292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:55.298269    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:20:55.501728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:55.501673    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" podStartSLOduration=5.501657724 podStartE2EDuration="5.501657724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:52.887238425 +0000 UTC m=+1741.761509314" watchObservedRunningTime="2026-11-02 20:20:55.501657724 +0000 UTC m=+1744.375928659"
Nov 02 20:20:56.415005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:56.414969    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:20:56.415370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:20:56.415036    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:21:03.685087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.685010    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:21:03.692888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.692865    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:21:03.693021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.692944    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.707679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.707648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.707822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.707696    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.707990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.707966    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t9dj\" (UniqueName: \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.708102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.708086    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.743919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.743887    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:21:03.809605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.809564    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t9dj\" (UniqueName: \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.809851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.809666    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.809851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.809777    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.809851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.809835    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.810716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.810690    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.833472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.833422    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a51b3f-serving-cert\""
Nov 02 20:21:03.833721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.833422    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\""
Nov 02 20:21:03.833721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.833555    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 20:21:03.841359 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:03.841339    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-serving-cert: secret "maas-api-e2e-shared-b-w1-a51b3f-serving-cert" not found
Nov 02 20:21:03.841487 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:03.841426    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls podName:3739c5da-cc9e-4017-a820-143420717b78 nodeName:}" failed. No retries permitted until 2026-11-02 20:21:04.341411295 +0000 UTC m=+1753.215682166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls") pod "maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" (UID: "3739c5da-cc9e-4017-a820-143420717b78") : secret "maas-api-e2e-shared-b-w1-a51b3f-serving-cert" not found
Nov 02 20:21:03.843606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.843586    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:03.884011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:03.883977    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t9dj\" (UniqueName: \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:04.420744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.420706    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:04.423371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.423340    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:04.608708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.608670    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:04.703604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.703570    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:21:04.993836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.993802    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:21:04.994704 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:21:04.994671    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3739c5da_cc9e_4017_a820_143420717b78.slice/crio-7b9ed2ea85edcfb53a4bb2dfcef2907096ebbf75e5f17917366414cc8efee8aa WatchSource:0}: Error finding container 7b9ed2ea85edcfb53a4bb2dfcef2907096ebbf75e5f17917366414cc8efee8aa: Status 404 returned error can't find the container with id 7b9ed2ea85edcfb53a4bb2dfcef2907096ebbf75e5f17917366414cc8efee8aa
Nov 02 20:21:04.996462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:04.996441    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:21:05.345747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.345661    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:21:05.345926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.345875    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" containerID="cri-o://e82158a28cf0dbbffa37e3a8de208d6e416480769f7cffd3c6e24d7035d871d2" gracePeriod=30
Nov 02 20:21:05.392698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.392655    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:21:05.392965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.392919    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" containerID="cri-o://8eaef030c6a79ee5c4e05ad2a22830b6659a3013a76ab504cfbd4d8f4a54fb8e" gracePeriod=30
Nov 02 20:21:05.397512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.397484    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:21:05.397688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.397669    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy" containerID="cri-o://387da1f747206c76ead1ad0f6731089201bddc799f9f28465264c5d90fb105dd" gracePeriod=30
Nov 02 20:21:05.401594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.401571    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:21:05.401771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.401752    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" containerID="cri-o://0cc5070f4285bb40706817fed7c953b9940dc811e9a7e90c28cf1dcebe7384e7" gracePeriod=30
Nov 02 20:21:05.818110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:05.818063    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" event={"ID":"3739c5da-cc9e-4017-a820-143420717b78","Type":"ContainerStarted","Data":"7b9ed2ea85edcfb53a4bb2dfcef2907096ebbf75e5f17917366414cc8efee8aa"}
Nov 02 20:21:06.348637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.348581    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:21:06.348904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.348852    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy" containerID="cri-o://8fc56ed6755f1c70e880e1dfb647a9dd6f05f0e516da39967c3a510dfa18c910" gracePeriod=30
Nov 02 20:21:06.354994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.354963    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:21:06.355181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.355159    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" containerID="cri-o://d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7" gracePeriod=30
Nov 02 20:21:06.360982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.360948    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:21:06.361200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.361174    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy" containerID="cri-o://177e5ba5e2a1f23719b098ba161f49351b42e408189a1de9683731d06ca45ba6" gracePeriod=30
Nov 02 20:21:06.393445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.393410    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:21:06.393629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:06.393608    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy" containerID="cri-o://969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a" gracePeriod=30
Nov 02 20:21:07.639111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:07.639069    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:21:07.639595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:07.639135    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:21:09.456056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:09.456013    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:21:09.456654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:09.456099    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:21:09.779700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:09.779612    5099 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:21:09.779700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:09.779676    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:21:10.371858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.371757    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:21:10.376497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.376472    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.434760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.434722    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:21:10.469829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.469370    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-602be312\"/\"openshift-service-ca.crt\""
Nov 02 20:21:10.469829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.469392    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-602be312\"/\"kube-root-ca.crt\""
Nov 02 20:21:10.469829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.469492    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-602be312\"/\"default-dockercfg-fc8t2\""
Nov 02 20:21:10.476716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.474757    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-602be312\"/\"e2e-lifecycle-model-602be312-kserve-self-signed-certs\""
Nov 02 20:21:10.506547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506505    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.506547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506570    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.506837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506614    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.506837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506671    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.506837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506816    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9fw\" (UniqueName: \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.506953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.506939    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.629835    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.629898    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.630053    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.630114    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.630240    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9fw\" (UniqueName: \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.630358    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.631442    5099 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.631485    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.632067    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.632337    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.646018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.630670    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.673768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.673512    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.673947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.673803    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:10.691422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:10.691317    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9fw\" (UniqueName: \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\") pod \"e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") " pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:11.062625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.055562    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:11.070681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.069506    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" event={"ID":"3739c5da-cc9e-4017-a820-143420717b78","Type":"ContainerStarted","Data":"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"}
Nov 02 20:21:11.070681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.070610    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:11.072854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.072829    5099 generic.go:356] "Generic (PLEG): container finished" podID="ac482a80-0e41-4747-b414-dc0f1506789f" containerID="0cc5070f4285bb40706817fed7c953b9940dc811e9a7e90c28cf1dcebe7384e7" exitCode=0
Nov 02 20:21:11.073140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.073121    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" event={"ID":"ac482a80-0e41-4747-b414-dc0f1506789f","Type":"ContainerDied","Data":"0cc5070f4285bb40706817fed7c953b9940dc811e9a7e90c28cf1dcebe7384e7"}
Nov 02 20:21:11.076528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.076291    5099 generic.go:356] "Generic (PLEG): container finished" podID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerID="e82158a28cf0dbbffa37e3a8de208d6e416480769f7cffd3c6e24d7035d871d2" exitCode=0
Nov 02 20:21:11.076883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.076722    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" event={"ID":"8f787339-a01a-4321-9acc-e4a485ff36fd","Type":"ContainerDied","Data":"e82158a28cf0dbbffa37e3a8de208d6e416480769f7cffd3c6e24d7035d871d2"}
Nov 02 20:21:11.079165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.079140    5099 generic.go:356] "Generic (PLEG): container finished" podID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerID="8eaef030c6a79ee5c4e05ad2a22830b6659a3013a76ab504cfbd4d8f4a54fb8e" exitCode=0
Nov 02 20:21:11.079401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.079379    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" event={"ID":"a89644dc-1644-4bec-89c1-131e2b4b988c","Type":"ContainerDied","Data":"8eaef030c6a79ee5c4e05ad2a22830b6659a3013a76ab504cfbd4d8f4a54fb8e"}
Nov 02 20:21:11.087524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.087459    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" event={"ID":"103abf05-c5fd-4b0a-9919-615b748b59f9","Type":"ContainerDied","Data":"387da1f747206c76ead1ad0f6731089201bddc799f9f28465264c5d90fb105dd"}
Nov 02 20:21:11.087883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.087838    5099 generic.go:356] "Generic (PLEG): container finished" podID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerID="387da1f747206c76ead1ad0f6731089201bddc799f9f28465264c5d90fb105dd" exitCode=0
Nov 02 20:21:11.088162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.088092    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7" event={"ID":"103abf05-c5fd-4b0a-9919-615b748b59f9","Type":"ContainerDied","Data":"4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75"}
Nov 02 20:21:11.088162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.088141    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5415c3bf3c6cffc463d4a7f8f1aafef811ada3593077f67a3614aacad4fc75"
Nov 02 20:21:11.089849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.089690    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:21:11.161804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.155308    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168475    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168525    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168575    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168626    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168684    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168741    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168775    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168848    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168876    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.168909    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169078    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169123    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169153    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169187    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169242    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169306    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\") pod \"103abf05-c5fd-4b0a-9919-615b748b59f9\" (UID: \"103abf05-c5fd-4b0a-9919-615b748b59f9\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.169354    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\") pod \"a89644dc-1644-4bec-89c1-131e2b4b988c\" (UID: \"a89644dc-1644-4bec-89c1-131e2b4b988c\") "
Nov 02 20:21:11.186240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.172178    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.198254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.194961    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.198254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.195052    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.198254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.195162    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.198254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.196949    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.201175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.200768    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.202490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.201347    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.202490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.201514    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.206315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.203758    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" podStartSLOduration=3.229896131 podStartE2EDuration="8.203738285s" totalImagesPullingTime="4.973842154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:11.196874305 +0000 UTC m=+1760.071145193" watchObservedRunningTime="2026-11-02 20:21:11.203738285 +0000 UTC m=+1760.078009175"
Nov 02 20:21:11.208821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.206487    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.212230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.209229    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "kube-api-access-nkbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.214719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.214027    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.220377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.220125    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.221911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.221875    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.236040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.223030    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.236040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.223423    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.236562 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.236538    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.242938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.242908    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo" pod "a89644dc-1644-4bec-89c1-131e2b4b988c" (UID: "a89644dc-1644-4bec-89c1-131e2b4b988c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.243147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.243124    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf" pod "103abf05-c5fd-4b0a-9919-615b748b59f9" (UID: "103abf05-c5fd-4b0a-9919-615b748b59f9"). InnerVolumeSpecName "kube-api-access-24jnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.267288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.267252    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:21:11.271173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.270517    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.271173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.270567    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.271173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.270623    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.271173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.270704    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.271173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271029    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271374    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271418    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271453    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271483    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271527    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271593    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\") pod \"8f787339-a01a-4321-9acc-e4a485ff36fd\" (UID: \"8f787339-a01a-4321-9acc-e4a485ff36fd\") "
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271619    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.271955    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272152    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbrz\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-kube-api-access-nkbrz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272171    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272179    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103abf05-c5fd-4b0a-9919-615b748b59f9-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272195    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jnf\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-kube-api-access-24jnf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272209    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f787339-a01a-4321-9acc-e4a485ff36fd-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272223    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272236    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272251    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272265    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272278    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272293    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272308    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272320    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272335    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272350    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272364    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272378    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272394    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272335    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272407    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272437    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a89644dc-1644-4bec-89c1-131e2b4b988c-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272453    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103abf05-c5fd-4b0a-9919-615b748b59f9-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.272469    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a89644dc-1644-4bec-89c1-131e2b4b988c-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.275394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.274118    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.277196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.275448    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.277196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.275657    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.282177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.282148    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh" pod "8f787339-a01a-4321-9acc-e4a485ff36fd" (UID: "8f787339-a01a-4321-9acc-e4a485ff36fd"). InnerVolumeSpecName "kube-api-access-mbwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.426762    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.426865    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.426904    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.426979    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.427095    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.427138    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.427292    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.427350    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.428873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.427586    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\") pod \"ac482a80-0e41-4747-b414-dc0f1506789f\" (UID: \"ac482a80-0e41-4747-b414-dc0f1506789f\") "
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.432474    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.433587    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.433799    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.434180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436453    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436477    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436491    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436505    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436517    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436530    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac482a80-0e41-4747-b414-dc0f1506789f-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436544    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f787339-a01a-4321-9acc-e4a485ff36fd-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436557    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436570    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwrh\" (UniqueName: \"kubernetes.io/projected/8f787339-a01a-4321-9acc-e4a485ff36fd-kube-api-access-mbwrh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.436584    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.441119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.437339    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.474093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.466245    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.474093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.467073    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.474093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.471995    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.476958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.476915    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c" pod "ac482a80-0e41-4747-b414-dc0f1506789f" (UID: "ac482a80-0e41-4747-b414-dc0f1506789f"). InnerVolumeSpecName "kube-api-access-tpz6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.538918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.538591    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.538918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.538625    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.538918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.538640    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac482a80-0e41-4747-b414-dc0f1506789f-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.538918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.538656    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac482a80-0e41-4747-b414-dc0f1506789f-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.538918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.538668    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz6c\" (UniqueName: \"kubernetes.io/projected/ac482a80-0e41-4747-b414-dc0f1506789f-kube-api-access-tpz6c\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.551295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.551162    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:21:11.574326 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:21:11.574264    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6319c39_77fd_4bdb_aaa1_a70cb4551c87.slice/crio-d4b9b2882f4a7d38ad9e54039dc0b2be382018d845dc4b0be9f62c871170125c WatchSource:0}: Error finding container d4b9b2882f4a7d38ad9e54039dc0b2be382018d845dc4b0be9f62c871170125c: Status 404 returned error can't find the container with id d4b9b2882f4a7d38ad9e54039dc0b2be382018d845dc4b0be9f62c871170125c
Nov 02 20:21:11.765337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.764749    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:21:11.818728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.818700    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:21:11.845936 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.845907    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.845996    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846052    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846097    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846152    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846225    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846298    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846347    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.846521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846447    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\") pod \"53d9649a-9e29-4297-9f23-e6c800efba87\" (UID: \"53d9649a-9e29-4297-9f23-e6c800efba87\") "
Nov 02 20:21:11.847009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.846847    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.847178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.847157    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.847278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.847186    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.847665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.847356    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.847665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.847532    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.847896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.847870    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.852070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.850771    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.852429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.852408    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "kube-api-access-6fdz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.854094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.854073    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.855666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.855632    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy" pod "53d9649a-9e29-4297-9f23-e6c800efba87" (UID: "53d9649a-9e29-4297-9f23-e6c800efba87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.948505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.948263    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.948505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.948333    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.948505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.948376    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.948505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.948410    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.948505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.948463    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.950019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.949591    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:11.950019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.949927    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.950338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.950263    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.950596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.950563    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.951109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.951055    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.951771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.951281    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.951771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.951347    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.951771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.951523    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\") pod \"8a2ad25a-302e-4c06-a539-1df4b92c5756\" (UID: \"8a2ad25a-302e-4c06-a539-1df4b92c5756\") "
Nov 02 20:21:11.952006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.951904    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952603    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952630    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdz2\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-kube-api-access-6fdz2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952644    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952660    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d9649a-9e29-4297-9f23-e6c800efba87-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952673    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952687    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d9649a-9e29-4297-9f23-e6c800efba87-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.952765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952700    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952872    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:11.953157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.952941    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953299    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953337    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953353    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d9649a-9e29-4297-9f23-e6c800efba87-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953366    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953382    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d9649a-9e29-4297-9f23-e6c800efba87-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.953437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.953395    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2ad25a-302e-4c06-a539-1df4b92c5756-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:11.954747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.954238    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:11.957349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:11.957328    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb" pod "8a2ad25a-302e-4c06-a539-1df4b92c5756" (UID: "8a2ad25a-302e-4c06-a539-1df4b92c5756"). InnerVolumeSpecName "kube-api-access-sdpwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:12.055823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.055760    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdpwb\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-kube-api-access-sdpwb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.055823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.055822    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.056022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.055833    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.056022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.055842    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2ad25a-302e-4c06-a539-1df4b92c5756-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.118632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.118600    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"
Nov 02 20:21:12.119837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.119807    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj" event={"ID":"8f787339-a01a-4321-9acc-e4a485ff36fd","Type":"ContainerDied","Data":"b5915b3cf2fda21ce588ef88e0ce4f8d7a234e0f4d8816782f48602efd9fa659"}
Nov 02 20:21:12.120070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.119880    5099 scope.go:122] "RemoveContainer" containerID="e82158a28cf0dbbffa37e3a8de208d6e416480769f7cffd3c6e24d7035d871d2"
Nov 02 20:21:12.127363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.127330    5099 generic.go:356] "Generic (PLEG): container finished" podID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerID="177e5ba5e2a1f23719b098ba161f49351b42e408189a1de9683731d06ca45ba6" exitCode=0
Nov 02 20:21:12.127486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.127434    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerDied","Data":"177e5ba5e2a1f23719b098ba161f49351b42e408189a1de9683731d06ca45ba6"}
Nov 02 20:21:12.131546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.131525    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"
Nov 02 20:21:12.131640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.131531    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb" event={"ID":"a89644dc-1644-4bec-89c1-131e2b4b988c","Type":"ContainerDied","Data":"8d631665c5af623acce4011019becf2565e32f5f6bc6f2bdfd65e52454e0d0c6"}
Nov 02 20:21:12.133719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.133695    5099 generic.go:356] "Generic (PLEG): container finished" podID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerID="969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a" exitCode=0
Nov 02 20:21:12.133827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.133814    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" event={"ID":"8a2ad25a-302e-4c06-a539-1df4b92c5756","Type":"ContainerDied","Data":"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"}
Nov 02 20:21:12.133920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.133854    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v" event={"ID":"8a2ad25a-302e-4c06-a539-1df4b92c5756","Type":"ContainerDied","Data":"c62f4939b7b0afbd2d4ab776a6ba1b8af536e9ad1844e0a8ad25e2c689b5be44"}
Nov 02 20:21:12.134010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.133939    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"
Nov 02 20:21:12.139256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.139221    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" event={"ID":"b6319c39-77fd-4bdb-aaa1-a70cb4551c87","Type":"ContainerStarted","Data":"b3eb25fcc9d18ab1c2bebae4a56d2d0623c7f9fd70fab0f9c18c677a2f98dcc7"}
Nov 02 20:21:12.139563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.139262    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" event={"ID":"b6319c39-77fd-4bdb-aaa1-a70cb4551c87","Type":"ContainerStarted","Data":"d4b9b2882f4a7d38ad9e54039dc0b2be382018d845dc4b0be9f62c871170125c"}
Nov 02 20:21:12.142072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.142044    5099 generic.go:356] "Generic (PLEG): container finished" podID="53d9649a-9e29-4297-9f23-e6c800efba87" containerID="d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7" exitCode=0
Nov 02 20:21:12.142180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.142151    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"
Nov 02 20:21:12.142739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.142714    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" event={"ID":"53d9649a-9e29-4297-9f23-e6c800efba87","Type":"ContainerDied","Data":"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"}
Nov 02 20:21:12.142926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.142907    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx" event={"ID":"53d9649a-9e29-4297-9f23-e6c800efba87","Type":"ContainerDied","Data":"6d25fcdb49ba59cc8711d839b8a76deeaad9299dfc8f10d21df91dd944de0289"}
Nov 02 20:21:12.145013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.144992    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g" event={"ID":"ac482a80-0e41-4747-b414-dc0f1506789f","Type":"ContainerDied","Data":"ab9d5e207c73799f268da026a91e5113f90d96e2c0d6ef5d9eed3e131e6c5fad"}
Nov 02 20:21:12.145115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.145086    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"
Nov 02 20:21:12.149542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.149527    5099 scope.go:122] "RemoveContainer" containerID="289af6130ca8086588b591c492813f9abd640806f9a1d7c918ce51862092ed0b"
Nov 02 20:21:12.151454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.151386    5099 generic.go:356] "Generic (PLEG): container finished" podID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerID="8fc56ed6755f1c70e880e1dfb647a9dd6f05f0e516da39967c3a510dfa18c910" exitCode=0
Nov 02 20:21:12.151525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.151493    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"
Nov 02 20:21:12.152408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.152156    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" event={"ID":"eba9eaf8-c2c4-4ad0-af03-418b98d3336c","Type":"ContainerDied","Data":"8fc56ed6755f1c70e880e1dfb647a9dd6f05f0e516da39967c3a510dfa18c910"}
Nov 02 20:21:12.154740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.154718    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:21:12.192719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.189030    5099 scope.go:122] "RemoveContainer" containerID="8eaef030c6a79ee5c4e05ad2a22830b6659a3013a76ab504cfbd4d8f4a54fb8e"
Nov 02 20:21:12.195532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.195446    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:21:12.206719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.206694    5099 scope.go:122] "RemoveContainer" containerID="969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"
Nov 02 20:21:12.218358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.218338    5099 scope.go:122] "RemoveContainer" containerID="969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"
Nov 02 20:21:12.218666 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:12.218644    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a\": container with ID starting with 969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a not found: ID does not exist" containerID="969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"
Nov 02 20:21:12.218844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.218680    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a"} err="failed to get container status \"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a\": rpc error: code = NotFound desc = could not find container \"969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a\": container with ID starting with 969223f39bbec723dfc4f415e53bc5e785ec4d513ad568e6b9ff978f3db89b5a not found: ID does not exist"
Nov 02 20:21:12.218844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.218705    5099 scope.go:122] "RemoveContainer" containerID="d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"
Nov 02 20:21:12.229081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.229060    5099 scope.go:122] "RemoveContainer" containerID="d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"
Nov 02 20:21:12.229414 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:12.229389    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7\": container with ID starting with d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7 not found: ID does not exist" containerID="d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"
Nov 02 20:21:12.229478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.229425    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7"} err="failed to get container status \"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7\": rpc error: code = NotFound desc = could not find container \"d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7\": container with ID starting with d5b054c771dceddf9f978459a9ff9c28d31ef3edf5ddc94aa1320405e8af81b7 not found: ID does not exist"
Nov 02 20:21:12.229478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.229449    5099 scope.go:122] "RemoveContainer" containerID="0cc5070f4285bb40706817fed7c953b9940dc811e9a7e90c28cf1dcebe7384e7"
Nov 02 20:21:12.257517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257488    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.257641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257589    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.257715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257642    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.257715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257681    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.257868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257731    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.257967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257865    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.257967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257899    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.257967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257912    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:12.257967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257925    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.258171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.257999    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.258171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258038    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.258171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258100    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.258171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258128    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.258171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258166    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258086    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258200    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\") pod \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\" (UID: \"eba9eaf8-c2c4-4ad0-af03-418b98d3336c\") "
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258227    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258316    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258354    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258407    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.258433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258419    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.258761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258477    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\") pod \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\" (UID: \"6369b245-aba9-46fc-b5dd-3f26e05e2b15\") "
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.259248    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.259276    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.259289    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.259302    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258420    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258586    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.259893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.258597    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.260417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.260390    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:12.260657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.260637    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.260895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.260872    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.261068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.261050    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.262571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.262518    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:21:12.262571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.262558    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.262922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.262892    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "kube-api-access-md2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:12.263178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.263155    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:12.263317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.263294    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy" pod "6369b245-aba9-46fc-b5dd-3f26e05e2b15" (UID: "6369b245-aba9-46fc-b5dd-3f26e05e2b15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:21:12.263694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.263666    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "kube-api-access-2chqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:12.264143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.264123    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token" pod "eba9eaf8-c2c4-4ad0-af03-418b98d3336c" (UID: "eba9eaf8-c2c4-4ad0-af03-418b98d3336c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:12.278931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.278889    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" podStartSLOduration=2.278875217 podStartE2EDuration="2.278875217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:12.278005255 +0000 UTC m=+1761.152276144" watchObservedRunningTime="2026-11-02 20:21:12.278875217 +0000 UTC m=+1761.153146104"
Nov 02 20:21:12.360472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360423    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360468    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chqj\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-kube-api-access-2chqj\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360485    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360499    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360513    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360526    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2sb\" (UniqueName: \"kubernetes.io/projected/6369b245-aba9-46fc-b5dd-3f26e05e2b15-kube-api-access-md2sb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360540    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360554    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360562    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369b245-aba9-46fc-b5dd-3f26e05e2b15-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360570    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360580    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360587    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360597    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.360744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.360611    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba9eaf8-c2c4-4ad0-af03-418b98d3336c-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:12.711980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.711937    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:21:12.719154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.719122    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-6lkgb"]
Nov 02 20:21:12.982966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:12.982878    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:21:13.000897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.000863    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zt4vj"]
Nov 02 20:21:13.067576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.067536    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:13.088822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.088766    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:21:13.095006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.094975    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-p2g7g"]
Nov 02 20:21:13.182639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.182599    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg" event={"ID":"eba9eaf8-c2c4-4ad0-af03-418b98d3336c","Type":"ContainerDied","Data":"c5e8286a2ddc8f85407c62024cbd00150713d049712616411cd8dea125eb0fc4"}
Nov 02 20:21:13.183046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.183028    5099 scope.go:122] "RemoveContainer" containerID="8fc56ed6755f1c70e880e1dfb647a9dd6f05f0e516da39967c3a510dfa18c910"
Nov 02 20:21:13.183346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.183331    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"
Nov 02 20:21:13.189260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.189227    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz" event={"ID":"6369b245-aba9-46fc-b5dd-3f26e05e2b15","Type":"ContainerDied","Data":"a6244bc46da443f3fbc71256d8376fb5ccf51df574822b1777eb8333dc04c32d"}
Nov 02 20:21:13.189260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.189249    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"
Nov 02 20:21:13.200701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.200677    5099 scope.go:122] "RemoveContainer" containerID="177e5ba5e2a1f23719b098ba161f49351b42e408189a1de9683731d06ca45ba6"
Nov 02 20:21:13.201567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.201363    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:21:13.205231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.205207    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fx9v7"]
Nov 02 20:21:13.262153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.262072    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" path="/var/lib/kubelet/pods/103abf05-c5fd-4b0a-9919-615b748b59f9/volumes"
Nov 02 20:21:13.262671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.262656    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" path="/var/lib/kubelet/pods/8f787339-a01a-4321-9acc-e4a485ff36fd/volumes"
Nov 02 20:21:13.263174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.263160    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" path="/var/lib/kubelet/pods/a89644dc-1644-4bec-89c1-131e2b4b988c/volumes"
Nov 02 20:21:13.263944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.263928    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" path="/var/lib/kubelet/pods/ac482a80-0e41-4747-b414-dc0f1506789f/volumes"
Nov 02 20:21:13.304612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.304570    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:21:13.310105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.310077    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-4cx5v"]
Nov 02 20:21:13.413536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.413496    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:21:13.417276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.417238    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-nkktx"]
Nov 02 20:21:13.521927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.521835    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:21:13.525493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.525464    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-htqzg"]
Nov 02 20:21:13.689136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.689105    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:21:13.692508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:13.692482    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-fnfzz"]
Nov 02 20:21:15.262814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:15.262749    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" path="/var/lib/kubelet/pods/53d9649a-9e29-4297-9f23-e6c800efba87/volumes"
Nov 02 20:21:15.263729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:15.263705    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" path="/var/lib/kubelet/pods/6369b245-aba9-46fc-b5dd-3f26e05e2b15/volumes"
Nov 02 20:21:15.264395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:15.264376    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" path="/var/lib/kubelet/pods/8a2ad25a-302e-4c06-a539-1df4b92c5756/volumes"
Nov 02 20:21:15.265472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:15.265453    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" path="/var/lib/kubelet/pods/eba9eaf8-c2c4-4ad0-af03-418b98d3336c/volumes"
Nov 02 20:21:16.107748 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:16.107707    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:21:18.201915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:18.201884    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:21:21.067864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:21.067817    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:21.081147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:21.081119    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:21.278314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:21.278283    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:21:31.605412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.605372    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:21:31.606429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606407    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606432    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606451    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606459    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606476    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606484    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606520    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.606527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606529    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606547    5099 cpu_manager.go:414] "containerMap: removing container" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606554    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606575    5099 cpu_manager.go:414] "containerMap: removing container" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606582    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606596    5099 cpu_manager.go:414] "containerMap: removing container" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606604    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606621    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606629    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606647    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy"
Nov 02 20:21:31.606762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606654    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606832    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f787339-a01a-4321-9acc-e4a485ff36fd" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606846    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606861    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6369b245-aba9-46fc-b5dd-3f26e05e2b15" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606872    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="103abf05-c5fd-4b0a-9919-615b748b59f9" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606885    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba9eaf8-c2c4-4ad0-af03-418b98d3336c" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606894    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d9649a-9e29-4297-9f23-e6c800efba87" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606904    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89644dc-1644-4bec-89c1-131e2b4b988c" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606913    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac482a80-0e41-4747-b414-dc0f1506789f" containerName="istio-proxy"
Nov 02 20:21:31.607084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.606924    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2ad25a-302e-4c06-a539-1df4b92c5756" containerName="istio-proxy"
Nov 02 20:21:31.613029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.613006    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.661129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.661088    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:21:31.670127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.670075    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.670294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.670137    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcz4n\" (UniqueName: \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.670294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.670261    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.670433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.670402    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.702707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.702673    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4dc8ca-serving-cert\""
Nov 02 20:21:31.771674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.771641    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.771885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.771682    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcz4n\" (UniqueName: \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.771885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.771852    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.772045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.772027    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.772168 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:31.772152    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-serving-cert: secret "maas-api-e2e-shared-a-w3-4dc8ca-serving-cert" not found
Nov 02 20:21:31.772250 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:31.772235    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls podName:43e1dd02-54db-420f-b605-9a7198bea27a nodeName:}" failed. No retries permitted until 2026-11-02 20:21:32.272205585 +0000 UTC m=+1781.146476453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls") pod "maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" (UID: "43e1dd02-54db-420f-b605-9a7198bea27a") : secret "maas-api-e2e-shared-a-w3-4dc8ca-serving-cert" not found
Nov 02 20:21:31.772588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.772564    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.774289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.774272    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:31.828766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:31.828733    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcz4n\" (UniqueName: \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:32.282511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:32.282473    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:32.285108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:32.285081    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:32.526709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:32.526676    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:32.911216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:32.911186    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:21:32.913911 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:21:32.913883    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e1dd02_54db_420f_b605_9a7198bea27a.slice/crio-308641cd533e82ed5c433a14fa977ca56633b18a7a07825365cdbefca037e40f WatchSource:0}: Error finding container 308641cd533e82ed5c433a14fa977ca56633b18a7a07825365cdbefca037e40f: Status 404 returned error can't find the container with id 308641cd533e82ed5c433a14fa977ca56633b18a7a07825365cdbefca037e40f
Nov 02 20:21:33.357424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:33.357331    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" event={"ID":"43e1dd02-54db-420f-b605-9a7198bea27a","Type":"ContainerStarted","Data":"308641cd533e82ed5c433a14fa977ca56633b18a7a07825365cdbefca037e40f"}
Nov 02 20:21:34.369432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:34.369387    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" event={"ID":"43e1dd02-54db-420f-b605-9a7198bea27a","Type":"ContainerStarted","Data":"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"}
Nov 02 20:21:34.369836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:34.369493    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:40.387311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:40.387272    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:21:40.495985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:40.495928    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" podStartSLOduration=8.57520211 podStartE2EDuration="9.495915076s" totalImagesPullingTime="920.712966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:34.492132196 +0000 UTC m=+1783.366403085" watchObservedRunningTime="2026-11-02 20:21:40.495915076 +0000 UTC m=+1789.370185964"
Nov 02 20:21:44.573091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.573054    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"]
Nov 02 20:21:44.578715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.578690    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.601078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.601042    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.601078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.601079    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.601275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.601181    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vnx\" (UniqueName: \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.630573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.630542    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"]
Nov 02 20:21:44.672562 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.672523    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:21:44.672729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.672600    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:21:44.702423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.702370    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.702638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.702523    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.702638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.702582    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vnx\" (UniqueName: \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.703291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.703260    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.705145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.705124    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.763937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.763904    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vnx\" (UniqueName: \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\") pod \"maas-api-revoke-sub-7432de73-4m5dr\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:44.891852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:44.891817    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:45.281848 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:21:45.281816    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291edc28_9a38_4f22_a0c3_5db00c2e1959.slice/crio-bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146 WatchSource:0}: Error finding container bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146: Status 404 returned error can't find the container with id bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146
Nov 02 20:21:45.285754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.285727    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"]
Nov 02 20:21:45.473915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.473880    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr" event={"ID":"291edc28-9a38-4f22-a0c3-5db00c2e1959","Type":"ContainerStarted","Data":"bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146"}
Nov 02 20:21:45.484807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.484767    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:21:45.488369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.488345    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.510715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.510684    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.510868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.510818    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.510925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.510864    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcgm\" (UniqueName: \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.511035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.511018    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.569068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.565547    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-5fc50d-serving-cert\""
Nov 02 20:21:45.571924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.571892    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:21:45.612774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.612734    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.613309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.612868    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcgm\" (UniqueName: \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.613309 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:45.612928    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-serving-cert: secret "maas-api-e2e-shared-a-w6-5fc50d-serving-cert" not found
Nov 02 20:21:45.613309 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:45.612997    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls podName:d5bb0cee-f735-4885-bc4a-f116c8f24186 nodeName:}" failed. No retries permitted until 2026-11-02 20:21:46.1129806 +0000 UTC m=+1794.987251492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls") pod "maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" (UID: "d5bb0cee-f735-4885-bc4a-f116c8f24186") : secret "maas-api-e2e-shared-a-w6-5fc50d-serving-cert" not found
Nov 02 20:21:45.613309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.613043    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.613309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.613189    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.613906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.613878    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.615671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.615646    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:45.669964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:45.669925    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcgm\" (UniqueName: \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:46.094401 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:21:46.094369    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:21:46.126273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.126239    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:46.128531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.128507    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:46.401312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.401274    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:46.489986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.489946    5099 generic.go:356] "Generic (PLEG): container finished" podID="291edc28-9a38-4f22-a0c3-5db00c2e1959" containerID="5bc5a5b8101c9711a7d447522fd1aa431cace89158e3bf1228c6936c24a3eff8" exitCode=0
Nov 02 20:21:46.490184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.490032    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr" event={"ID":"291edc28-9a38-4f22-a0c3-5db00c2e1959","Type":"ContainerDied","Data":"5bc5a5b8101c9711a7d447522fd1aa431cace89158e3bf1228c6936c24a3eff8"}
Nov 02 20:21:46.829737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:46.829705    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:21:46.831079 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:21:46.831044    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bb0cee_f735_4885_bc4a_f116c8f24186.slice/crio-dca4b5aef0f18ce1e093f8a6d3e2e48cb5d363029198d17356e9f9ec7ebaef60 WatchSource:0}: Error finding container dca4b5aef0f18ce1e093f8a6d3e2e48cb5d363029198d17356e9f9ec7ebaef60: Status 404 returned error can't find the container with id dca4b5aef0f18ce1e093f8a6d3e2e48cb5d363029198d17356e9f9ec7ebaef60
Nov 02 20:21:47.527980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.527935    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" event={"ID":"d5bb0cee-f735-4885-bc4a-f116c8f24186","Type":"ContainerStarted","Data":"dca4b5aef0f18ce1e093f8a6d3e2e48cb5d363029198d17356e9f9ec7ebaef60"}
Nov 02 20:21:47.790832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.790801    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:47.975175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.975136    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\" (UniqueName: \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\") pod \"291edc28-9a38-4f22-a0c3-5db00c2e1959\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") "
Nov 02 20:21:47.975175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.975176    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\") pod \"291edc28-9a38-4f22-a0c3-5db00c2e1959\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") "
Nov 02 20:21:47.975711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.975296    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\") pod \"291edc28-9a38-4f22-a0c3-5db00c2e1959\" (UID: \"291edc28-9a38-4f22-a0c3-5db00c2e1959\") "
Nov 02 20:21:47.975711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.975673    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca" pod "291edc28-9a38-4f22-a0c3-5db00c2e1959" (UID: "291edc28-9a38-4f22-a0c3-5db00c2e1959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:21:47.977455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.977432    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx" pod "291edc28-9a38-4f22-a0c3-5db00c2e1959" (UID: "291edc28-9a38-4f22-a0c3-5db00c2e1959"). InnerVolumeSpecName "kube-api-access-59vnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:21:47.977553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:47.977482    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca" pod "291edc28-9a38-4f22-a0c3-5db00c2e1959" (UID: "291edc28-9a38-4f22-a0c3-5db00c2e1959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:21:48.076680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.076590    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59vnx\" (UniqueName: \"kubernetes.io/projected/291edc28-9a38-4f22-a0c3-5db00c2e1959-kube-api-access-59vnx\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:48.076680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.076623    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:48.076680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.076633    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/291edc28-9a38-4f22-a0c3-5db00c2e1959-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:21:48.580808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.577686    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr" event={"ID":"291edc28-9a38-4f22-a0c3-5db00c2e1959","Type":"ContainerDied","Data":"bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146"}
Nov 02 20:21:48.580808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.577749    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc4e9e9b6a0301f4dd7fc4a633d21a7c5c8e4af9372324286a5dfa9cc171a146"
Nov 02 20:21:48.580808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.577900    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"
Nov 02 20:21:48.592382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.592337    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" event={"ID":"d5bb0cee-f735-4885-bc4a-f116c8f24186","Type":"ContainerStarted","Data":"7480d60bc2e09a637b11bc47fffcbd0927c6b86105942d52f2d560f627193d77"}
Nov 02 20:21:48.592549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.592442    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:48.716694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:48.716645    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" podStartSLOduration=2.8459219559999998 podStartE2EDuration="3.716631727s" totalImagesPullingTime="870.709771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:48.714389626 +0000 UTC m=+1797.588660517" watchObservedRunningTime="2026-11-02 20:21:48.716631727 +0000 UTC m=+1797.590902614"
Nov 02 20:21:51.743152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:51.743126    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:21:51.761709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:51.761681    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:21:53.256132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.254764    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": read tcp 10.132.0.2:35048->10.132.0.164:15021: read: connection reset by peer" start-of-body=
Nov 02 20:21:53.256132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.254848    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": read tcp 10.132.0.2:35048->10.132.0.164:15021: read: connection reset by peer"
Nov 02 20:21:53.487659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.487571    5099 scope.go:122] "RemoveContainer" containerID="2c35a2c9384935415d1c828213a11b86c31a4ef3fba2e2d2393178238c2c8d95"
Nov 02 20:21:53.498153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.497911    5099 scope.go:122] "RemoveContainer" containerID="9ff485d2cc23871ba06a19d0a451160372c742ba31e090fa00bee9a978981536"
Nov 02 20:21:53.507263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.507232    5099 scope.go:122] "RemoveContainer" containerID="387da1f747206c76ead1ad0f6731089201bddc799f9f28465264c5d90fb105dd"
Nov 02 20:21:53.522190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.522172    5099 scope.go:122] "RemoveContainer" containerID="388df67fea9590150f00836f7a8cd8601d7796ce5967e3fd1511e689e2aa8b6c"
Nov 02 20:21:53.639269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.639240    5099 generic.go:356] "Generic (PLEG): container finished" podID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerID="d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04" exitCode=0
Nov 02 20:21:53.639446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.639320    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerDied","Data":"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"}
Nov 02 20:21:53.639821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.639776    5099 scope.go:122] "RemoveContainer" containerID="d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"
Nov 02 20:21:53.641388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.641363    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:53.641460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.641429    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:53.641508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:53.641460    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:21:54.608845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:54.608814    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:21:54.676324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:54.676286    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerStarted","Data":"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"}
Nov 02 20:21:55.324749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:55.324713    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:21:55.324949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:55.324900    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 20:21:55.324995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:55.324945    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 20:21:56.324513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:56.324480    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 20:21:56.324922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:56.324536    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 20:21:57.324978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:57.324891    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 20:21:57.324978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:57.324944    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 20:21:58.324972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:58.324937    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 20:21:58.325460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:58.324998    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 20:21:59.325156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:59.325119    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 20:21:59.325579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:21:59.325178    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 20:22:00.329666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:00.329630    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:22:00.330197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:00.330049    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:22:00.330834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:00.330814    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:22:13.454403 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.454366    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7183-1f1c1b"]
Nov 02 20:22:13.455082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.455065    5099 cpu_manager.go:414] "containerMap: removing container" podUID="291edc28-9a38-4f22-a0c3-5db00c2e1959" containerName="revoke-keys"
Nov 02 20:22:13.455146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.455084    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291edc28-9a38-4f22-a0c3-5db00c2e1959" containerName="revoke-keys"
Nov 02 20:22:13.455269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.455251    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="291edc28-9a38-4f22-a0c3-5db00c2e1959" containerName="revoke-keys"
Nov 02 20:22:13.457774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.457755    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:13.481373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.481319    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sflmq\" (UniqueName: \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\") pod \"test-curl-7183-1f1c1b\" (UID: \"d7540c88-0fe1-4a37-8b08-5f811eb01045\") " pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:13.509135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.509100    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-1f1c1b"]
Nov 02 20:22:13.551676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.551633    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-28p54\""
Nov 02 20:22:13.582836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.582766    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sflmq\" (UniqueName: \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\") pod \"test-curl-7183-1f1c1b\" (UID: \"d7540c88-0fe1-4a37-8b08-5f811eb01045\") " pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:13.637046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.637010    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sflmq\" (UniqueName: \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\") pod \"test-curl-7183-1f1c1b\" (UID: \"d7540c88-0fe1-4a37-8b08-5f811eb01045\") " pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:13.770629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:13.770532    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:14.223029 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:14.222984    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7540c88_0fe1_4a37_8b08_5f811eb01045.slice/crio-364c053ca689f453ee4b2928137c48d7f6bf3265900839a730dbeae0492d36a8 WatchSource:0}: Error finding container 364c053ca689f453ee4b2928137c48d7f6bf3265900839a730dbeae0492d36a8: Status 404 returned error can't find the container with id 364c053ca689f453ee4b2928137c48d7f6bf3265900839a730dbeae0492d36a8
Nov 02 20:22:14.223214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:14.223191    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-1f1c1b"]
Nov 02 20:22:14.910490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:14.910454    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-1f1c1b" event={"ID":"d7540c88-0fe1-4a37-8b08-5f811eb01045","Type":"ContainerStarted","Data":"364c053ca689f453ee4b2928137c48d7f6bf3265900839a730dbeae0492d36a8"}
Nov 02 20:22:16.046350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:16.046314    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"]
Nov 02 20:22:16.094730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:16.094681    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83492e8e-rcc94"]
Nov 02 20:22:16.106899 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:16.106856    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:22:17.267728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:17.267648    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee603e47-f86a-4fc9-b654-78ee4f3dfbd0" path="/var/lib/kubelet/pods/ee603e47-f86a-4fc9-b654-78ee4f3dfbd0/volumes"
Nov 02 20:22:17.935333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:17.935296    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-1f1c1b" event={"ID":"d7540c88-0fe1-4a37-8b08-5f811eb01045","Type":"ContainerStarted","Data":"4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3"}
Nov 02 20:22:18.042295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:18.042229    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7183-1f1c1b" podStartSLOduration=2.367059835 podStartE2EDuration="5.042210114s" totalImagesPullingTime="2.675150279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:18.040807435 +0000 UTC m=+1826.915078324" watchObservedRunningTime="2026-11-02 20:22:18.042210114 +0000 UTC m=+1826.916481003"
Nov 02 20:22:19.426067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.426032    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7183-1f1c1b"]
Nov 02 20:22:19.430129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.430098    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7183-1f1c1b"]
Nov 02 20:22:19.573056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.573014    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7183-a5cf13"]
Nov 02 20:22:19.573721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.573702    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" containerName="test-curl-7183-1f1c1b"
Nov 02 20:22:19.573721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.573720    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" containerName="test-curl-7183-1f1c1b"
Nov 02 20:22:19.573896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.573863    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" containerName="test-curl-7183-1f1c1b"
Nov 02 20:22:19.576998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.576974    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:19.627456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.627426    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-a5cf13"]
Nov 02 20:22:19.694287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.694206    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfrz\" (UniqueName: \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\") pod \"test-curl-7183-a5cf13\" (UID: \"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6\") " pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:19.795835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.795768    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfrz\" (UniqueName: \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\") pod \"test-curl-7183-a5cf13\" (UID: \"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6\") " pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:19.849771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.849734    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfrz\" (UniqueName: \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\") pod \"test-curl-7183-a5cf13\" (UID: \"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6\") " pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:19.889766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.889732    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:19.961023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:19.958045    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7183-1f1c1b" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" containerName="test-curl-7183-1f1c1b" containerID="cri-o://4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3" gracePeriod=2
Nov 02 20:22:20.006346 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:20.006292    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-1f1c1b\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:20.261289 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:20.261244    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a9cbf9d_dc3d_4b1f_8c85_9ff59275eda6.slice/crio-42ee144755f47ee46a7e549a967d6dd7f391e4952c65630a842a166a39a20ab3 WatchSource:0}: Error finding container 42ee144755f47ee46a7e549a967d6dd7f391e4952c65630a842a166a39a20ab3: Status 404 returned error can't find the container with id 42ee144755f47ee46a7e549a967d6dd7f391e4952c65630a842a166a39a20ab3
Nov 02 20:22:20.261451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:20.261340    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-a5cf13"]
Nov 02 20:22:20.968828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:20.968771    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-a5cf13" event={"ID":"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6","Type":"ContainerStarted","Data":"722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2"}
Nov 02 20:22:20.969229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:20.968843    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-a5cf13" event={"ID":"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6","Type":"ContainerStarted","Data":"42ee144755f47ee46a7e549a967d6dd7f391e4952c65630a842a166a39a20ab3"}
Nov 02 20:22:21.017343 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:21.017298    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-1f1c1b\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:21.125560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:21.125505    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7183-a5cf13" podStartSLOduration=2.12548973 podStartE2EDuration="2.12548973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:21.122459237 +0000 UTC m=+1829.996730124" watchObservedRunningTime="2026-11-02 20:22:21.12548973 +0000 UTC m=+1829.999760618"
Nov 02 20:22:21.304722 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:21.304636    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-1f1c1b\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:22.223194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.223124    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:22.331304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.331272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\" (UniqueName: \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\") pod \"d7540c88-0fe1-4a37-8b08-5f811eb01045\" (UID: \"d7540c88-0fe1-4a37-8b08-5f811eb01045\") "
Nov 02 20:22:22.333437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.333409    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq" pod "d7540c88-0fe1-4a37-8b08-5f811eb01045" (UID: "d7540c88-0fe1-4a37-8b08-5f811eb01045"). InnerVolumeSpecName "kube-api-access-sflmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:22.433474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.433439    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sflmq\" (UniqueName: \"kubernetes.io/projected/d7540c88-0fe1-4a37-8b08-5f811eb01045-kube-api-access-sflmq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:22.519606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.519523    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7183-a5cf13"]
Nov 02 20:22:22.523993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.523964    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7183-a5cf13"]
Nov 02 20:22:22.667441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.667409    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7183-38a6dd"]
Nov 02 20:22:22.668112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.668096    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" containerName="test-curl-7183-a5cf13"
Nov 02 20:22:22.668166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.668113    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" containerName="test-curl-7183-a5cf13"
Nov 02 20:22:22.668257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.668247    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" containerName="test-curl-7183-a5cf13"
Nov 02 20:22:22.670508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.670490    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:22.721078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.721046    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-38a6dd"]
Nov 02 20:22:22.838234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.838142    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9x4v\" (UniqueName: \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\") pod \"test-curl-7183-38a6dd\" (UID: \"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0\") " pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:22.939633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.939598    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9x4v\" (UniqueName: \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\") pod \"test-curl-7183-38a6dd\" (UID: \"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0\") " pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:22.986008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.985977    5099 generic.go:356] "Generic (PLEG): container finished" podID="d7540c88-0fe1-4a37-8b08-5f811eb01045" containerID="4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3" exitCode=137
Nov 02 20:22:22.986178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.986016    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:22.986178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.986065    5099 scope.go:122] "RemoveContainer" containerID="4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3"
Nov 02 20:22:22.986325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.986297    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7183-a5cf13" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" containerName="test-curl-7183-a5cf13" containerID="cri-o://722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2" gracePeriod=2
Nov 02 20:22:22.993495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.993469    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9x4v\" (UniqueName: \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\") pod \"test-curl-7183-38a6dd\" (UID: \"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0\") " pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:22.996536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.996515    5099 scope.go:122] "RemoveContainer" containerID="4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3"
Nov 02 20:22:22.996847 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:22.996824    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3\": container with ID starting with 4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3 not found: ID does not exist" containerID="4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3"
Nov 02 20:22:22.996924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:22.996857    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3"} err="failed to get container status \"4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3\": rpc error: code = NotFound desc = could not find container \"4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3\": container with ID starting with 4b78f9fab72887b19ce4e58fbd7976bc916cad1db446797ed9b2d45af0636ce3 not found: ID does not exist"
Nov 02 20:22:23.034523 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:23.034485    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-a5cf13\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:23.082685 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:23.082652    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-1f1c1b\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" pod="opendatahub/test-curl-7183-1f1c1b"
Nov 02 20:22:23.131008 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:23.130967    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-a5cf13\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:23.259822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:23.259776    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7540c88-0fe1-4a37-8b08-5f811eb01045" path="/var/lib/kubelet/pods/d7540c88-0fe1-4a37-8b08-5f811eb01045/volumes"
Nov 02 20:22:23.282653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:23.282625    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:23.758071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:23.757993    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-38a6dd"]
Nov 02 20:22:23.996113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:23.996072    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-38a6dd" event={"ID":"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0","Type":"ContainerStarted","Data":"556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd"}
Nov 02 20:22:23.996113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:23.996118    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-38a6dd" event={"ID":"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0","Type":"ContainerStarted","Data":"8a6acd3387088c0d389d3270ee8e9988e41b09428e651c836edcbbc14fb1921d"}
Nov 02 20:22:24.102981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:24.102883    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7183-38a6dd" podStartSLOduration=2.102869045 podStartE2EDuration="2.102869045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:24.101419515 +0000 UTC m=+1832.975690403" watchObservedRunningTime="2026-11-02 20:22:24.102869045 +0000 UTC m=+1832.977139932"
Nov 02 20:22:25.238057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.238032    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:25.387069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.387035    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\" (UniqueName: \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\") pod \"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6\" (UID: \"3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6\") "
Nov 02 20:22:25.389301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.389272    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz" pod "3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" (UID: "3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6"). InnerVolumeSpecName "kube-api-access-9sfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:25.441249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.441213    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7183-38a6dd"]
Nov 02 20:22:25.444845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.444812    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7183-38a6dd"]
Nov 02 20:22:25.489460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.489427    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfrz\" (UniqueName: \"kubernetes.io/projected/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6-kube-api-access-9sfrz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:25.584849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.584817    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7183-6c73a5"]
Nov 02 20:22:25.585509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.585495    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" containerName="test-curl-7183-38a6dd"
Nov 02 20:22:25.585553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.585510    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" containerName="test-curl-7183-38a6dd"
Nov 02 20:22:25.585654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.585645    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" containerName="test-curl-7183-38a6dd"
Nov 02 20:22:25.588033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.588014    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:25.639986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.639905    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-6c73a5"]
Nov 02 20:22:25.692701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.692671    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmr7h\" (UniqueName: \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\") pod \"test-curl-7183-6c73a5\" (UID: \"95d01309-c39a-414a-ad7e-7441b6c05ed2\") " pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:25.793833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.793764    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmr7h\" (UniqueName: \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\") pod \"test-curl-7183-6c73a5\" (UID: \"95d01309-c39a-414a-ad7e-7441b6c05ed2\") " pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:25.847752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.847721    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmr7h\" (UniqueName: \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\") pod \"test-curl-7183-6c73a5\" (UID: \"95d01309-c39a-414a-ad7e-7441b6c05ed2\") " pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:25.900336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:25.900263    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:26.019511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.019471    5099 generic.go:356] "Generic (PLEG): container finished" podID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" containerID="722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2" exitCode=137
Nov 02 20:22:26.019691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.019557    5099 scope.go:122] "RemoveContainer" containerID="722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2"
Nov 02 20:22:26.019761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.019689    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-a5cf13"
Nov 02 20:22:26.019905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.019878    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7183-38a6dd" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" containerName="test-curl-7183-38a6dd" containerID="cri-o://556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd" gracePeriod=2
Nov 02 20:22:26.034677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.034654    5099 scope.go:122] "RemoveContainer" containerID="722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2"
Nov 02 20:22:26.034991 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:26.034969    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2\": container with ID starting with 722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2 not found: ID does not exist" containerID="722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2"
Nov 02 20:22:26.035059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.035004    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2"} err="failed to get container status \"722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2\": rpc error: code = NotFound desc = could not find container \"722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2\": container with ID starting with 722560978c7c148c07b708e583fbf35c27822b162cbba0b421dd768140387ea2 not found: ID does not exist"
Nov 02 20:22:26.068356 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:26.068321    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-38a6dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:26.116765 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:26.116731    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-38a6dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:26.276036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:26.276005    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7183-6c73a5"]
Nov 02 20:22:26.276550 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:26.276526    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d01309_c39a_414a_ad7e_7441b6c05ed2.slice/crio-1aa7e90cab7ea170673d705ac776e4db200d78680be627b24b6e464d29d173df WatchSource:0}: Error finding container 1aa7e90cab7ea170673d705ac776e4db200d78680be627b24b6e464d29d173df: Status 404 returned error can't find the container with id 1aa7e90cab7ea170673d705ac776e4db200d78680be627b24b6e464d29d173df
Nov 02 20:22:27.029436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:27.029399    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-6c73a5" event={"ID":"95d01309-c39a-414a-ad7e-7441b6c05ed2","Type":"ContainerStarted","Data":"a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9"}
Nov 02 20:22:27.029436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:27.029447    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7183-6c73a5" event={"ID":"95d01309-c39a-414a-ad7e-7441b6c05ed2","Type":"ContainerStarted","Data":"1aa7e90cab7ea170673d705ac776e4db200d78680be627b24b6e464d29d173df"}
Nov 02 20:22:27.077883 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:27.077841    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-38a6dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:27.184146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:27.184096    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7183-6c73a5" podStartSLOduration=2.184083441 podStartE2EDuration="2.184083441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:27.183070346 +0000 UTC m=+1836.057341233" watchObservedRunningTime="2026-11-02 20:22:27.184083441 +0000 UTC m=+1836.058354329"
Nov 02 20:22:27.259321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:27.259288    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6" path="/var/lib/kubelet/pods/3a9cbf9d-dc3d-4b1f-8c85-9ff59275eda6/volumes"
Nov 02 20:22:28.287101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.287030    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:28.327525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.327491    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\" (UniqueName: \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\") pod \"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0\" (UID: \"3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0\") "
Nov 02 20:22:28.329556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.329523    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v" pod "3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" (UID: "3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0"). InnerVolumeSpecName "kube-api-access-d9x4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:28.429891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.429854    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9x4v\" (UniqueName: \"kubernetes.io/projected/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0-kube-api-access-d9x4v\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:28.621869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.621831    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7183-6c73a5"]
Nov 02 20:22:28.627321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:28.627291    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7183-6c73a5"]
Nov 02 20:22:29.069558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.069469    5099 generic.go:356] "Generic (PLEG): container finished" podID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" containerID="556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd" exitCode=137
Nov 02 20:22:29.069558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.069511    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:29.069802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.069569    5099 scope.go:122] "RemoveContainer" containerID="556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd"
Nov 02 20:22:29.069869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.069808    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7183-6c73a5" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" containerName="test-curl-7183-6c73a5" containerID="cri-o://a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9" gracePeriod=2
Nov 02 20:22:29.080643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.080628    5099 scope.go:122] "RemoveContainer" containerID="556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd"
Nov 02 20:22:29.080940 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:29.080923    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd\": container with ID starting with 556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd not found: ID does not exist" containerID="556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd"
Nov 02 20:22:29.080998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.080947    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd"} err="failed to get container status \"556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd\": rpc error: code = NotFound desc = could not find container \"556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd\": container with ID starting with 556328bfee0f9a0421a189acdf26685fd6c2d0f54a241a22e582c87bd68ae9fd not found: ID does not exist"
Nov 02 20:22:29.118220 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:29.118183    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:29.166544 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:29.166498    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-38a6dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" pod="opendatahub/test-curl-7183-38a6dd"
Nov 02 20:22:29.214622 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:29.214579    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:29.259101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.259069    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0" path="/var/lib/kubelet/pods/3ad2b8e8-2db4-4fd9-9fb4-a47854a69fb0/volumes"
Nov 02 20:22:29.643204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.643167    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:22:29.643876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.643860    5099 cpu_manager.go:414] "containerMap: removing container" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" containerName="test-curl-7183-6c73a5"
Nov 02 20:22:29.643939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.643876    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" containerName="test-curl-7183-6c73a5"
Nov 02 20:22:29.644003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.643994    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" containerName="test-curl-7183-6c73a5"
Nov 02 20:22:29.647202 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.647182    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.706270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.706196    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7dcb03-openshift-default-dockercfg-mr2wr\""
Nov 02 20:22:29.747443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747401    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747473    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747506    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747555    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747597    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747640    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747700    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747768    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.747884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.747854    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtd82\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.759768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.759734    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.849301    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.849435    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.849494    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.849903    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.850031    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.850115    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtd82\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.850274    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.850336    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.850665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.850381    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.853797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.851958    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.853797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.853730    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.854671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.854651    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.855601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.854931    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.855752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.855267    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.857564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.856505    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.904172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.896485    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.910882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.910764    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.911061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.910948    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtd82\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:29.962271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:29.962238    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:30.386875 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:30.386841    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c1eb63_b8e9_4ce0_9f29_1e758e742c5c.slice/crio-9cef7ef0a6c5655722af6626a51f1fc6699832bd4d5ebd1ac9cd813115164a92 WatchSource:0}: Error finding container 9cef7ef0a6c5655722af6626a51f1fc6699832bd4d5ebd1ac9cd813115164a92: Status 404 returned error can't find the container with id 9cef7ef0a6c5655722af6626a51f1fc6699832bd4d5ebd1ac9cd813115164a92
Nov 02 20:22:30.387530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:30.387510    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:22:30.389474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:30.389440    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:30.389558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:30.389506    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:30.389558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:30.389533    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:31.101680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.101650    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerStarted","Data":"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"}
Nov 02 20:22:31.102022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.101740    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerStarted","Data":"9cef7ef0a6c5655722af6626a51f1fc6699832bd4d5ebd1ac9cd813115164a92"}
Nov 02 20:22:31.260380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.258394    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podStartSLOduration=2.258377495 podStartE2EDuration="2.258377495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:31.257809507 +0000 UTC m=+1840.132080398" watchObservedRunningTime="2026-11-02 20:22:31.258377495 +0000 UTC m=+1840.132648384"
Nov 02 20:22:31.326104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.326035    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:31.391820 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:31.387065    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:31.391820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.387754    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\" (UniqueName: \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\") pod \"95d01309-c39a-414a-ad7e-7441b6c05ed2\" (UID: \"95d01309-c39a-414a-ad7e-7441b6c05ed2\") "
Nov 02 20:22:31.401847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.399622    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h" pod "95d01309-c39a-414a-ad7e-7441b6c05ed2" (UID: "95d01309-c39a-414a-ad7e-7441b6c05ed2"). InnerVolumeSpecName "kube-api-access-dmr7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:22:31.489679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.489641    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmr7h\" (UniqueName: \"kubernetes.io/projected/95d01309-c39a-414a-ad7e-7441b6c05ed2-kube-api-access-dmr7h\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:22:31.962528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.962472    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:31.968491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:31.968462    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:32.016731 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:32.016690    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:32.111386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.111353    5099 generic.go:356] "Generic (PLEG): container finished" podID="95d01309-c39a-414a-ad7e-7441b6c05ed2" containerID="a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9" exitCode=137
Nov 02 20:22:32.111851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.111420    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:32.111851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.111455    5099 scope.go:122] "RemoveContainer" containerID="a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9"
Nov 02 20:22:32.112029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.112007    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:32.113242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.113222    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:22:32.123255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.123233    5099 scope.go:122] "RemoveContainer" containerID="a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9"
Nov 02 20:22:32.123541 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:32.123519    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9\": container with ID starting with a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9 not found: ID does not exist" containerID="a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9"
Nov 02 20:22:32.123635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.123551    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9"} err="failed to get container status \"a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9\": rpc error: code = NotFound desc = could not find container \"a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9\": container with ID starting with a25ca49875f528e471168fcf8a0b07c0164415f1837f67a2604bf698461dc2f9 not found: ID does not exist"
Nov 02 20:22:32.250422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.250338    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:22:32.255143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.255118    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320661    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320709    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320736    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320798    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320845    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320873    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320946    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46pd\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.320990    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.321095    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.322504    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c76017-openshift-default-dockercfg-p9q56\""
Nov 02 20:22:32.332004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.322855    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:22:32.348859 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:32.348759    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:32.422691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422657    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.422940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422708    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.422940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422743    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.422940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422841    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.422940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422891    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.422940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.422929    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423027    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46pd\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423100    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423223    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423247    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423309    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423543    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423582    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.423705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.423685    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.425282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.425265    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.425572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.425557    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.476626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.476596    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.476956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.476929    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46pd\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:32.529125 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:32.528131    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7183-6c73a5\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" pod="opendatahub/test-curl-7183-6c73a5"
Nov 02 20:22:32.575812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:32.572084    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:33.068370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.068334    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:22:33.102297 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:33.102266    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019e565a_a379_4576_b830_48d8f0252824.slice/crio-3fe506c1787e8b119eab86518c88b693ca88e7080db5b2de7a67d5671f86db37 WatchSource:0}: Error finding container 3fe506c1787e8b119eab86518c88b693ca88e7080db5b2de7a67d5671f86db37: Status 404 returned error can't find the container with id 3fe506c1787e8b119eab86518c88b693ca88e7080db5b2de7a67d5671f86db37
Nov 02 20:22:33.105029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.104996    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:33.105237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.105067    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:33.105237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.105098    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:33.121441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.121202    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerStarted","Data":"3fe506c1787e8b119eab86518c88b693ca88e7080db5b2de7a67d5671f86db37"}
Nov 02 20:22:33.260271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:33.260231    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d01309-c39a-414a-ad7e-7441b6c05ed2" path="/var/lib/kubelet/pods/95d01309-c39a-414a-ad7e-7441b6c05ed2/volumes"
Nov 02 20:22:34.130014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:34.129982    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerStarted","Data":"206164b41ae845d734b404a112bde69dadcf1716f60ea500fea0804dfc0abadd"}
Nov 02 20:22:34.293149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:34.293088    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podStartSLOduration=2.2930695500000002 podStartE2EDuration="2.29306955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:34.288597859 +0000 UTC m=+1843.162868748" watchObservedRunningTime="2026-11-02 20:22:34.29306955 +0000 UTC m=+1843.167340441"
Nov 02 20:22:34.582866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:34.579510    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:34.596917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:34.596888    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:35.138037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:35.138006    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:35.138992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:35.138973    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:22:38.072810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.070155    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:22:38.077088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.075690    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.120604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.120237    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j72tz\" (UniqueName: \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.120604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.120448    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.146810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.131204    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:22:38.210179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.210144    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7dcb03\""
Nov 02 20:22:38.330857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.234301    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.330857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.329650    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j72tz\" (UniqueName: \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.330857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.236522    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7dcb03-dockercfg-gwqnv\""
Nov 02 20:22:38.330857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.235311    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.358285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.358021    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:22:38.372247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.372220    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.390898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.390832    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j72tz\" (UniqueName: \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\") pod \"payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.400078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.400045    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:22:38.430563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.430532    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.430825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.430769    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnj6\" (UniqueName: \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.469146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.469110    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:38.532212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.532171    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.532402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.532348    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnj6\" (UniqueName: \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.533169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.533066    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.590645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.589460    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnj6\" (UniqueName: \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\") pod \"payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.716888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.716853    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:38.986933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:38.985538    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:22:39.282028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.281936    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" event={"ID":"fad19e3a-d94a-481c-8608-31ae0115387a","Type":"ContainerStarted","Data":"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"}
Nov 02 20:22:39.282477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.282444    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" event={"ID":"fad19e3a-d94a-481c-8608-31ae0115387a","Type":"ContainerStarted","Data":"12d8c60c7c93218cb545d03977f8dffe01c2cd734a42ffef5a490ea380fdc1ee"}
Nov 02 20:22:39.282590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.282503    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:39.393588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.393539    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" podStartSLOduration=2.393524956 podStartE2EDuration="2.393524956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:39.391900645 +0000 UTC m=+1848.266171534" watchObservedRunningTime="2026-11-02 20:22:39.393524956 +0000 UTC m=+1848.267795895"
Nov 02 20:22:39.418297 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:39.418056    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffab4fc_381b_4f15_862b_393e830cb029.slice/crio-7a6ec76b3676d0b03fc1a1cb0250b27e9e03685d0e09ef15b9d8da4ff625cd01 WatchSource:0}: Error finding container 7a6ec76b3676d0b03fc1a1cb0250b27e9e03685d0e09ef15b9d8da4ff625cd01: Status 404 returned error can't find the container with id 7a6ec76b3676d0b03fc1a1cb0250b27e9e03685d0e09ef15b9d8da4ff625cd01
Nov 02 20:22:39.419645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.419616    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:22:39.601220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.601182    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:22:39.605168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.605146    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.663811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.663062    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5efe19-openshift-default-dockercfg-fh76j\""
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681531    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681607    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681642    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681678    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681696    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681732    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681775    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681845    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvsm\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.682128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.681894    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.689763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.689733    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:22:39.783128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783084    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvsm\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783159    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783263    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783354    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783531    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783616    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783648    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783715    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783798    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783816    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.783946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.783854    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.784061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.784043    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.784224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.784201    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.784514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.784485    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.786085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.786052    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.786276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.786255    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.859859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.837094    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.859859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.841445    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvsm\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\") pod \"e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") " pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:39.925402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:39.925065    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:40.293976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.293934    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" event={"ID":"affab4fc-381b-4f15-862b-393e830cb029","Type":"ContainerStarted","Data":"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"}
Nov 02 20:22:40.294430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.293997    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" event={"ID":"affab4fc-381b-4f15-862b-393e830cb029","Type":"ContainerStarted","Data":"7a6ec76b3676d0b03fc1a1cb0250b27e9e03685d0e09ef15b9d8da4ff625cd01"}
Nov 02 20:22:40.294430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.294027    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:40.419636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.419571    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" podStartSLOduration=2.419554421 podStartE2EDuration="2.419554421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:40.416267996 +0000 UTC m=+1849.290538886" watchObservedRunningTime="2026-11-02 20:22:40.419554421 +0000 UTC m=+1849.293825319"
Nov 02 20:22:40.478607 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:40.474085    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982ee009_cc01_4529_bd61_abbefa04fec6.slice/crio-1009c8daa5084ea33887581811c4f3d4351acc47a0850891b8775c60af260a5c WatchSource:0}: Error finding container 1009c8daa5084ea33887581811c4f3d4351acc47a0850891b8775c60af260a5c: Status 404 returned error can't find the container with id 1009c8daa5084ea33887581811c4f3d4351acc47a0850891b8775c60af260a5c
Nov 02 20:22:40.478607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.477585    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:22:40.480152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.480114    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:40.480251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.480211    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:40.480288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:40.480268    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:41.304378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:41.304341    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerStarted","Data":"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"}
Nov 02 20:22:41.304758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:41.304396    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerStarted","Data":"1009c8daa5084ea33887581811c4f3d4351acc47a0850891b8775c60af260a5c"}
Nov 02 20:22:41.427876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:41.427826    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podStartSLOduration=2.427811751 podStartE2EDuration="2.427811751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:41.424800589 +0000 UTC m=+1850.299071490" watchObservedRunningTime="2026-11-02 20:22:41.427811751 +0000 UTC m=+1850.302082636"
Nov 02 20:22:41.926730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:41.926697    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:41.932940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:41.932915    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:42.410945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:42.410909    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:42.434568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:42.434393    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:22:45.844655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.844608    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:22:45.881875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.880323    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:45.939807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.932100    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:22:45.951884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.949035    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c76017\""
Nov 02 20:22:45.979809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.969110    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c76017-dockercfg-tgq29\""
Nov 02 20:22:45.984747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.981769    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52bh\" (UniqueName: \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:45.984747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:45.982146    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.034965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.034932    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:22:46.092259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.087148    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.092259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.087251    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c52bh\" (UniqueName: \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.092259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.088870    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.092259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.090267    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.092259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.092111    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:22:46.191514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.190958    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.191514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.191068    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zx8h\" (UniqueName: \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.203347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.203230    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52bh\" (UniqueName: \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\") pod \"payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.228827 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:22:46.228553    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:22:46.270997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.270958    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:46.295016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.292937    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zx8h\" (UniqueName: \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.295016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.293279    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.298824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.298699    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.364891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.363007    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zx8h\" (UniqueName: \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\") pod \"payload-processing-e2e-api-a-c76017-f5894596c-p2qsl\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") " pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.530456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.530365    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:46.657831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.657659    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:22:46.668433 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:46.668387    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d390e7_494a_4580_978d_c3fc3e62b572.slice/crio-5d2e6d7bf10e6cac098738f298ea81e409cf35d2010a3da813f27b781e5c9d8a WatchSource:0}: Error finding container 5d2e6d7bf10e6cac098738f298ea81e409cf35d2010a3da813f27b781e5c9d8a: Status 404 returned error can't find the container with id 5d2e6d7bf10e6cac098738f298ea81e409cf35d2010a3da813f27b781e5c9d8a
Nov 02 20:22:46.920876 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:46.920844    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccd97a7_719b_4832_b85e_e7c0728f276a.slice/crio-93ba7075fd80b493c50a9865097bd6338081d13ef06a99697d5c5fd35b515f59 WatchSource:0}: Error finding container 93ba7075fd80b493c50a9865097bd6338081d13ef06a99697d5c5fd35b515f59: Status 404 returned error can't find the container with id 93ba7075fd80b493c50a9865097bd6338081d13ef06a99697d5c5fd35b515f59
Nov 02 20:22:46.921205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:46.920945    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:22:47.460421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.460321    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" event={"ID":"07d390e7-494a-4580-978d-c3fc3e62b572","Type":"ContainerStarted","Data":"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"}
Nov 02 20:22:47.460421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.460406    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:47.460714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.460432    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" event={"ID":"07d390e7-494a-4580-978d-c3fc3e62b572","Type":"ContainerStarted","Data":"5d2e6d7bf10e6cac098738f298ea81e409cf35d2010a3da813f27b781e5c9d8a"}
Nov 02 20:22:47.462211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.462178    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" event={"ID":"9ccd97a7-719b-4832-b85e-e7c0728f276a","Type":"ContainerStarted","Data":"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"}
Nov 02 20:22:47.462349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.462234    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" event={"ID":"9ccd97a7-719b-4832-b85e-e7c0728f276a","Type":"ContainerStarted","Data":"93ba7075fd80b493c50a9865097bd6338081d13ef06a99697d5c5fd35b515f59"}
Nov 02 20:22:47.462349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.462317    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:47.464338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.464311    5099 generic.go:356] "Generic (PLEG): container finished" podID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerID="94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a" exitCode=0
Nov 02 20:22:47.464427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.464375    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerDied","Data":"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"}
Nov 02 20:22:47.464728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.464715    5099 scope.go:122] "RemoveContainer" containerID="94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"
Nov 02 20:22:47.466352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.466323    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:47.466454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.466405    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:47.466519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.466458    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:47.573968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.573814    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" podStartSLOduration=2.57379365 podStartE2EDuration="2.57379365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:47.570158073 +0000 UTC m=+1856.444428963" watchObservedRunningTime="2026-11-02 20:22:47.57379365 +0000 UTC m=+1856.448064537"
Nov 02 20:22:47.847895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:47.846741    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" podStartSLOduration=2.846721726 podStartE2EDuration="2.846721726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:47.842985437 +0000 UTC m=+1856.717256328" watchObservedRunningTime="2026-11-02 20:22:47.846721726 +0000 UTC m=+1856.720992615"
Nov 02 20:22:48.562822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.562238    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:22:48.618770 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.618732    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:48.643704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.643668    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:22:48.684251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.684222    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5efe19-dockercfg-kd58t\""
Nov 02 20:22:48.684591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.684559    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerStarted","Data":"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"}
Nov 02 20:22:48.684916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.684892    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5efe19\""
Nov 02 20:22:48.711815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.708995    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:48.711815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.709151    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwppq\" (UniqueName: \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:48.900860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.839766    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:48.922402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.901423    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwppq\" (UniqueName: \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:48.922649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:48.880137    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:49.015355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.005855    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:22:49.044978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.044639    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.047759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.047626    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:22:49.068075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.061743    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwppq\" (UniqueName: \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\") pod \"payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:49.135761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.135728    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlv5x\" (UniqueName: \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.135939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.135919    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.243851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.241703    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlv5x\" (UniqueName: \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.243851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.241942    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.243851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.243031    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.285454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.285416    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:49.311241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.311214    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlv5x\" (UniqueName: \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\") pod \"payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.396668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.396635    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:49.597079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.597041    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:22:49.597967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.597943    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:49.598068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.597990    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:49.712362 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:49.712329    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5e35f5_c084_45e2_8f5e_4f6f119a9201.slice/crio-59b82f7ea0418ae748971191d492fe3ad2161212004e54518da5e0ae1ef8f854 WatchSource:0}: Error finding container 59b82f7ea0418ae748971191d492fe3ad2161212004e54518da5e0ae1ef8f854: Status 404 returned error can't find the container with id 59b82f7ea0418ae748971191d492fe3ad2161212004e54518da5e0ae1ef8f854
Nov 02 20:22:49.712515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.712493    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:22:49.854812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:49.854055    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:22:50.298129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.298099    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:22:50.425269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.425233    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:22:50.596853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.596763    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:50.596853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.596833    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:50.686232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.686197    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" event={"ID":"ab5e35f5-c084-45e2-8f5e-4f6f119a9201","Type":"ContainerStarted","Data":"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"}
Nov 02 20:22:50.686744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.686247    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" event={"ID":"ab5e35f5-c084-45e2-8f5e-4f6f119a9201","Type":"ContainerStarted","Data":"59b82f7ea0418ae748971191d492fe3ad2161212004e54518da5e0ae1ef8f854"}
Nov 02 20:22:50.686744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.686493    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:22:50.688236 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.688207    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" event={"ID":"78992bde-8c03-4475-a5b5-49cf6dc0a258","Type":"ContainerStarted","Data":"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"}
Nov 02 20:22:50.688352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.688258    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" event={"ID":"78992bde-8c03-4475-a5b5-49cf6dc0a258","Type":"ContainerStarted","Data":"4f043f0e2be33e21960cd8e7dba89f054290df3e90730357c9cf31df7a2c8ef4"}
Nov 02 20:22:50.688424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.688404    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:22:50.793687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.793638    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" podStartSLOduration=2.793623669 podStartE2EDuration="2.793623669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:50.792921622 +0000 UTC m=+1859.667192510" watchObservedRunningTime="2026-11-02 20:22:50.793623669 +0000 UTC m=+1859.667894557"
Nov 02 20:22:50.910203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:50.908978    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" podStartSLOduration=2.908948482 podStartE2EDuration="2.908948482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:50.906025039 +0000 UTC m=+1859.780295929" watchObservedRunningTime="2026-11-02 20:22:50.908948482 +0000 UTC m=+1859.783219370"
Nov 02 20:22:51.314897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:51.310118    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:22:51.597097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:51.597008    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:51.597097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:51.597067    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:52.599390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:52.599343    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:52.600048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:52.600012    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:53.621806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:53.606715    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:53.621806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:53.606835    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:54.510988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:54.510934    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:54.602399 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:54.602360    5099 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 20:22:54.603386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:54.603349    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 20:22:55.179776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.179730    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:55.212848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.212027    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 20:22:55.261887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.261842    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:55.295405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.285448    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:55.388722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.388675    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:55.418940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.418896    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:22:55.514414 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.514322    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:22:55.867865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.828926    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:55.867865 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.842896    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:22:55.887332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.887302    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-40425f-openshift-default-dockercfg-hlnj9\""
Nov 02 20:22:55.953317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.953273    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:22:55.992049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:55.992011    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:22:56.021703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.021662    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022056    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022244    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022429    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022572    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022681    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.022843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.022826    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvpp\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.023018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.023003    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.023852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.023821    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.129607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.129515    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.130465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.130427    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.131122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.131100    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.131628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.131610    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.132578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.132557    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.133113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.133092    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.133592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.133572    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.134501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.134480    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvpp\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.133500    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.131520    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.134387    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.132987    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.131005    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.135545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.135526    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.136508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.136485    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.166073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.163524    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.214664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.214621    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvpp\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.221219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.220315    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\") pod \"e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") " pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.221219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.221022    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:56.856451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.856345    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:22:56.891714 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:22:56.891684    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5653aa_ceeb_42b5_9801_33527250cc56.slice/crio-11b2880f2a1da13f54822c84c0ed408d5262bdb545d29adf6ff313f9e3fc3f4d WatchSource:0}: Error finding container 11b2880f2a1da13f54822c84c0ed408d5262bdb545d29adf6ff313f9e3fc3f4d: Status 404 returned error can't find the container with id 11b2880f2a1da13f54822c84c0ed408d5262bdb545d29adf6ff313f9e3fc3f4d
Nov 02 20:22:56.894540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.894507    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:56.894614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.894579    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:56.894614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:56.894606    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:22:57.021050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.021010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerStarted","Data":"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"}
Nov 02 20:22:57.021246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.021076    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerStarted","Data":"11b2880f2a1da13f54822c84c0ed408d5262bdb545d29adf6ff313f9e3fc3f4d"}
Nov 02 20:22:57.133625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.133547    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podStartSLOduration=2.133526907 podStartE2EDuration="2.133526907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:57.131502431 +0000 UTC m=+1866.005773319" watchObservedRunningTime="2026-11-02 20:22:57.133526907 +0000 UTC m=+1866.007797796"
Nov 02 20:22:57.222020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.221989    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:57.223945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.223914    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:22:57.224079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:57.223979    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:22:58.226729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:58.226698    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:58.664269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:58.664234    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:22:58.664442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:58.664320    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:22:59.036538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:59.036455    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:22:59.037516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:22:59.037486    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:23:01.702093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:01.702056    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:23:01.702494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:01.702143    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:23:03.705805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.702562    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:23:03.722808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.716865    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:03.755804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.754964    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:23:03.883352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.883316    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-40425f-dockercfg-xt2bm\""
Nov 02 20:23:03.883598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.883491    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-40425f\""
Nov 02 20:23:03.897950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.890658    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xz2\" (UniqueName: \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:03.897950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.890773    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:03.900651 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.900536    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:23:03.911519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.911490    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:23:03.911664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.911604    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:03.991995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.991908    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xz2\" (UniqueName: \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:03.992175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.992019    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:03.992175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.992057    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:03.992175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.992159    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfqf\" (UniqueName: \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:03.993148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:03.993127    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:04.045872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.045841    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xz2\" (UniqueName: \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\") pod \"payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:04.093866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.093825    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:04.094034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.093918    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfqf\" (UniqueName: \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:04.094536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.094515    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:04.147692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.147659    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfqf\" (UniqueName: \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\") pod \"payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:04.226826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.226778    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:04.253441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.253407    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:04.613003 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:23:04.612974    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8331cf1_415f_4198_b038_77b035431d10.slice/crio-94ca53d9bf63afc0c218e7e1ff0df85d4b503cf50ef9208942600d6357f9ae29 WatchSource:0}: Error finding container 94ca53d9bf63afc0c218e7e1ff0df85d4b503cf50ef9208942600d6357f9ae29: Status 404 returned error can't find the container with id 94ca53d9bf63afc0c218e7e1ff0df85d4b503cf50ef9208942600d6357f9ae29
Nov 02 20:23:04.613638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.613586    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:23:04.663042 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:23:04.663009    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c41a76_cfc6_4b45_b9f7_570a1d2634db.slice/crio-b2300ac55358756b5b52f07f1679d2907ddd724efb4c85df23da35b60fd74b65 WatchSource:0}: Error finding container b2300ac55358756b5b52f07f1679d2907ddd724efb4c85df23da35b60fd74b65: Status 404 returned error can't find the container with id b2300ac55358756b5b52f07f1679d2907ddd724efb4c85df23da35b60fd74b65
Nov 02 20:23:04.664346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:04.664325    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:23:05.089795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.089697    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" event={"ID":"46c41a76-cfc6-4b45-b9f7-570a1d2634db","Type":"ContainerStarted","Data":"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"}
Nov 02 20:23:05.089795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.089757    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" event={"ID":"46c41a76-cfc6-4b45-b9f7-570a1d2634db","Type":"ContainerStarted","Data":"b2300ac55358756b5b52f07f1679d2907ddd724efb4c85df23da35b60fd74b65"}
Nov 02 20:23:05.090332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.089803    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:05.091685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.091660    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" event={"ID":"e8331cf1-415f-4198-b038-77b035431d10","Type":"ContainerStarted","Data":"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"}
Nov 02 20:23:05.091818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.091705    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" event={"ID":"e8331cf1-415f-4198-b038-77b035431d10","Type":"ContainerStarted","Data":"94ca53d9bf63afc0c218e7e1ff0df85d4b503cf50ef9208942600d6357f9ae29"}
Nov 02 20:23:05.091818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.091753    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:05.198895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:05.198846    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" podStartSLOduration=2.198828908 podStartE2EDuration="2.198828908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:05.198314939 +0000 UTC m=+1874.072585832" watchObservedRunningTime="2026-11-02 20:23:05.198828908 +0000 UTC m=+1874.073099795"
Nov 02 20:23:13.041715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:13.041651    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" podStartSLOduration=10.04163607 podStartE2EDuration="10.04163607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:05.314607141 +0000 UTC m=+1874.188878055" watchObservedRunningTime="2026-11-02 20:23:13.04163607 +0000 UTC m=+1881.915906959"
Nov 02 20:23:13.044976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:13.044945    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"]
Nov 02 20:23:13.091466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:13.091433    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed0efb2-cndnf"]
Nov 02 20:23:13.260795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:13.260748    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3062eb-3ca7-4dd1-92df-2176f38352cd" path="/var/lib/kubelet/pods/ce3062eb-3ca7-4dd1-92df-2176f38352cd/volumes"
Nov 02 20:23:16.095894 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:16.095851    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:23:16.105671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:16.105649    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:23:16.105850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:16.105726    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:23:41.923908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:41.923872    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7186-c124dd"]
Nov 02 20:23:41.927383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:41.927363    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:41.977450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:41.977418    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7186-c124dd"]
Nov 02 20:23:42.018655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.018612    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-28p54\""
Nov 02 20:23:42.058306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.058272    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbhs\" (UniqueName: \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\") pod \"test-curl-7186-c124dd\" (UID: \"c351b353-e91b-4320-a286-1c5575dd20b4\") " pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:42.159521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.159485    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbhs\" (UniqueName: \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\") pod \"test-curl-7186-c124dd\" (UID: \"c351b353-e91b-4320-a286-1c5575dd20b4\") " pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:42.213164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.213090    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbhs\" (UniqueName: \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\") pod \"test-curl-7186-c124dd\" (UID: \"c351b353-e91b-4320-a286-1c5575dd20b4\") " pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:42.240700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.240674    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:42.611975 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:23:42.611659    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc351b353_e91b_4320_a286_1c5575dd20b4.slice/crio-6a6dd4d8ac2f9b0a559c67f40db828e3af84c51d9be3bea5ef7339a7dd7e6980 WatchSource:0}: Error finding container 6a6dd4d8ac2f9b0a559c67f40db828e3af84c51d9be3bea5ef7339a7dd7e6980: Status 404 returned error can't find the container with id 6a6dd4d8ac2f9b0a559c67f40db828e3af84c51d9be3bea5ef7339a7dd7e6980
Nov 02 20:23:42.612927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:42.612896    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7186-c124dd"]
Nov 02 20:23:43.476001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:43.475969    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7186-c124dd" event={"ID":"c351b353-e91b-4320-a286-1c5575dd20b4","Type":"ContainerStarted","Data":"43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c"}
Nov 02 20:23:43.476409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:43.476010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7186-c124dd" event={"ID":"c351b353-e91b-4320-a286-1c5575dd20b4","Type":"ContainerStarted","Data":"6a6dd4d8ac2f9b0a559c67f40db828e3af84c51d9be3bea5ef7339a7dd7e6980"}
Nov 02 20:23:43.591133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:43.591070    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7186-c124dd" podStartSLOduration=2.591052409 podStartE2EDuration="2.591052409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:43.589131551 +0000 UTC m=+1912.463402455" watchObservedRunningTime="2026-11-02 20:23:43.591052409 +0000 UTC m=+1912.465323299"
Nov 02 20:23:44.972500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:44.968083    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7186-c124dd"]
Nov 02 20:23:45.023835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.017356    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7186-c124dd"]
Nov 02 20:23:45.493019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.492969    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7186-c124dd" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" containerName="test-curl-7186-c124dd" containerID="cri-o://43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c" gracePeriod=2
Nov 02 20:23:45.820189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.820113    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"]
Nov 02 20:23:45.821028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.821005    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" containerName="test-curl-7186-c124dd"
Nov 02 20:23:45.821082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.821032    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" containerName="test-curl-7186-c124dd"
Nov 02 20:23:45.821215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.821204    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" containerName="test-curl-7186-c124dd"
Nov 02 20:23:45.823855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.823838    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.831670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.831649    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddxf\" (UniqueName: \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.831838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.831811    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.831897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.831860    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.877758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.877727    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"]
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.933861    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.934037    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.935356    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddxf\" (UniqueName: \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.935541    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.935590    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:45.943809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:45.936879    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:46.001452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:46.000259    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:46.066281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:46.059241    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddxf\" (UniqueName: \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\") pod \"maas-api-revoke-keys-e2e-mt-602be312-d66bp\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:46.120863 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:46.120828    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:23:46.137223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:46.137197    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:46.523673 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:23:46.523645    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5195e7d6_2f83_450f_9d96_5b88bd22b02c.slice/crio-5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b WatchSource:0}: Error finding container 5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b: Status 404 returned error can't find the container with id 5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b
Nov 02 20:23:46.524455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:46.524422    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"]
Nov 02 20:23:47.515769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.515744    5099 generic.go:356] "Generic (PLEG): container finished" podID="5195e7d6-2f83-450f-9d96-5b88bd22b02c" containerID="72d0baad3729ec129a1d25225df841e41eb1f7788bf76ef4698c40854d586541" exitCode=0
Nov 02 20:23:47.516182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.515819    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp" event={"ID":"5195e7d6-2f83-450f-9d96-5b88bd22b02c","Type":"ContainerDied","Data":"72d0baad3729ec129a1d25225df841e41eb1f7788bf76ef4698c40854d586541"}
Nov 02 20:23:47.516182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.515906    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp" event={"ID":"5195e7d6-2f83-450f-9d96-5b88bd22b02c","Type":"ContainerStarted","Data":"5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b"}
Nov 02 20:23:47.757194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.757168    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:47.785833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.785805    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\" (UniqueName: \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\") pod \"c351b353-e91b-4320-a286-1c5575dd20b4\" (UID: \"c351b353-e91b-4320-a286-1c5575dd20b4\") "
Nov 02 20:23:47.788022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.787997    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs" pod "c351b353-e91b-4320-a286-1c5575dd20b4" (UID: "c351b353-e91b-4320-a286-1c5575dd20b4"). InnerVolumeSpecName "kube-api-access-zvbhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:47.887428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:47.887386    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbhs\" (UniqueName: \"kubernetes.io/projected/c351b353-e91b-4320-a286-1c5575dd20b4-kube-api-access-zvbhs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:48.525123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.525088    5099 generic.go:356] "Generic (PLEG): container finished" podID="c351b353-e91b-4320-a286-1c5575dd20b4" containerID="43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c" exitCode=137
Nov 02 20:23:48.525578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.525130    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:48.525578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.525181    5099 scope.go:122] "RemoveContainer" containerID="43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c"
Nov 02 20:23:48.536465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.536445    5099 scope.go:122] "RemoveContainer" containerID="43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c"
Nov 02 20:23:48.536794 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:48.536763    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c\": container with ID starting with 43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c not found: ID does not exist" containerID="43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c"
Nov 02 20:23:48.536853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.536803    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c"} err="failed to get container status \"43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c\": rpc error: code = NotFound desc = could not find container \"43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c\": container with ID starting with 43bafbbcb14777c5a99cd7f3c516a8a2e53b92eb706c07036d66e429b2bbde7c not found: ID does not exist"
Nov 02 20:23:48.583507 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:48.583470    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7186-c124dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:48.691586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.691559    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:48.741185 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:48.741150    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7186-c124dd\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" pod="opendatahub/test-curl-7186-c124dd"
Nov 02 20:23:48.801708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.801604    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\") pod \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") "
Nov 02 20:23:48.801875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.801715    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\") pod \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") "
Nov 02 20:23:48.801930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.801907    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\" (UniqueName: \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\") pod \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\" (UID: \"5195e7d6-2f83-450f-9d96-5b88bd22b02c\") "
Nov 02 20:23:48.802106 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.802077    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca" pod "5195e7d6-2f83-450f-9d96-5b88bd22b02c" (UID: "5195e7d6-2f83-450f-9d96-5b88bd22b02c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:23:48.802520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.802497    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:48.804015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.803991    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca" pod "5195e7d6-2f83-450f-9d96-5b88bd22b02c" (UID: "5195e7d6-2f83-450f-9d96-5b88bd22b02c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:23:48.804306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.804280    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf" pod "5195e7d6-2f83-450f-9d96-5b88bd22b02c" (UID: "5195e7d6-2f83-450f-9d96-5b88bd22b02c"). InnerVolumeSpecName "kube-api-access-9ddxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:23:48.903706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.903667    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddxf\" (UniqueName: \"kubernetes.io/projected/5195e7d6-2f83-450f-9d96-5b88bd22b02c-kube-api-access-9ddxf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:48.903706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:48.903706    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5195e7d6-2f83-450f-9d96-5b88bd22b02c-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:23:49.260367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:49.260329    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c351b353-e91b-4320-a286-1c5575dd20b4" path="/var/lib/kubelet/pods/c351b353-e91b-4320-a286-1c5575dd20b4/volumes"
Nov 02 20:23:49.535977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:49.535901    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"
Nov 02 20:23:49.535977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:49.535946    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp" event={"ID":"5195e7d6-2f83-450f-9d96-5b88bd22b02c","Type":"ContainerDied","Data":"5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b"}
Nov 02 20:23:49.536405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:49.535993    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4d43bf3606452fa0852c451a38849329d7688403b9107bc4638e08a749266b"
Nov 02 20:23:54.084862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:54.084828    5099 scope.go:122] "RemoveContainer" containerID="8b120e642c00c4444d924069dd8e6546fe016467163cf3d659af9e9d554847ea"
Nov 02 20:23:54.096046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:54.096026    5099 scope.go:122] "RemoveContainer" containerID="62ec56e1ccc9c1ccf80173c1e16f510659f7cbd2c8c663437eda46da407d61c9"
Nov 02 20:23:54.105887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:54.105866    5099 scope.go:122] "RemoveContainer" containerID="d1ed4d83f31d065ea02b997b10a634627f158825d3dd4157c6368ed732d087f1"
Nov 02 20:23:54.115585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:54.115561    5099 scope.go:122] "RemoveContainer" containerID="0a1b29ec036fc9848247853698c77853c15330813de2b65465497ee26206b680"
Nov 02 20:23:57.379591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.379561    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:23:57.380453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.380435    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5195e7d6-2f83-450f-9d96-5b88bd22b02c" containerName="revoke-keys"
Nov 02 20:23:57.380453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.380453    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5195e7d6-2f83-450f-9d96-5b88bd22b02c" containerName="revoke-keys"
Nov 02 20:23:57.380615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.380604    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5195e7d6-2f83-450f-9d96-5b88bd22b02c" containerName="revoke-keys"
Nov 02 20:23:57.384271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.384250    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.436934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.436906    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:23:57.478641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.478602    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5efe19-serving-cert\""
Nov 02 20:23:57.535891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.535833    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.536213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.536188    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.536473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.536452    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnqj\" (UniqueName: \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.536760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.536735    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.641918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.641819    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.643042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.643002    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.650919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.646951    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.651195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.651172    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.651504 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:57.651413    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-serving-cert: secret "maas-api-e2e-shared-b-w3-5efe19-serving-cert" not found
Nov 02 20:23:57.651683 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:23:57.651672    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls podName:c205fbaa-0c0a-417a-962c-98faaff9d146 nodeName:}" failed. No retries permitted until 2026-11-02 20:23:58.151650056 +0000 UTC m=+1927.025920925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" (UID: "c205fbaa-0c0a-417a-962c-98faaff9d146") : secret "maas-api-e2e-shared-b-w3-5efe19-serving-cert" not found
Nov 02 20:23:57.658864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.658825    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnqj\" (UniqueName: \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.698801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.693549    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:57.726983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:57.726953    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnqj\" (UniqueName: \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:58.191114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:58.191072    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:58.193526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:58.193500    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:58.298107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:58.298074    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:23:58.670259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:58.670202    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:23:58.672313 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:23:58.672286    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc205fbaa_0c0a_417a_962c_98faaff9d146.slice/crio-f4ae6745b02c3356eb071f0bacf48e238ce613f8b31dd158eff00cf7792d3e62 WatchSource:0}: Error finding container f4ae6745b02c3356eb071f0bacf48e238ce613f8b31dd158eff00cf7792d3e62: Status 404 returned error can't find the container with id f4ae6745b02c3356eb071f0bacf48e238ce613f8b31dd158eff00cf7792d3e62
Nov 02 20:23:59.620716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:59.620681    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" event={"ID":"c205fbaa-0c0a-417a-962c-98faaff9d146","Type":"ContainerStarted","Data":"f4ae6745b02c3356eb071f0bacf48e238ce613f8b31dd158eff00cf7792d3e62"}
Nov 02 20:23:59.782436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:59.782375    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:23:59.897697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:23:59.897575    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:24:00.630465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:00.630432    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" event={"ID":"c205fbaa-0c0a-417a-962c-98faaff9d146","Type":"ContainerStarted","Data":"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"}
Nov 02 20:24:00.630680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:00.630662    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:24:00.743832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:00.743755    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" podStartSLOduration=2.778057532 podStartE2EDuration="3.743737149s" totalImagesPullingTime="965.679617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:00.737929467 +0000 UTC m=+1929.612200355" watchObservedRunningTime="2026-11-02 20:24:00.743737149 +0000 UTC m=+1929.618008038"
Nov 02 20:24:04.782725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:04.782673    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing" containerID="cri-o://7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f" gracePeriod=25
Nov 02 20:24:04.899347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:04.899283    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing" containerID="cri-o://e94aa3f8f8f9f2f10d393b40176bc7f77f09e52e9259b534eb1507a93eb3d075" gracePeriod=25
Nov 02 20:24:06.707531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:06.707497    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:24:08.896969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:08.896911    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:24:08.898028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:08.897995    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:24:09.282864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:09.282268    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:09.289467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:09.288403    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:09.383919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:09.383867    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:09.420981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:09.420917    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 02 20:24:09.514107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:09.514057    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:10.238115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:10.238069    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:10.306815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:10.303865    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:10.306815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:10.304907    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:24:10.438805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:10.435886    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:10.438805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:10.436402    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:24:11.140702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.140449    5099 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:11.179265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.179215    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:11.206074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.206028    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:11.304952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.304873    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:24:11.307736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.307704    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:11.399870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.394392    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:24:11.441289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.411547    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 02 20:24:11.441289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.440298    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:11.802311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.800910    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:24:11.802974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.802948    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:24:11.806505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:11.806479    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:24:15.519258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.519221    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:24:15.524332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.524314    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.573435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.573399    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-40425f-serving-cert\""
Nov 02 20:24:15.601761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.601717    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.602385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.601966    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prn56\" (UniqueName: \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.602385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.602090    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.602385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.602144    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.704146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.704058    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.704426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.704401    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.704824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.704801    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn56\" (UniqueName: \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.704997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.704978    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.705946 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:24:15.705919    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-serving-cert: secret "maas-api-e2e-shared-b-w6-40425f-serving-cert" not found
Nov 02 20:24:15.706160 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:24:15.706115    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls podName:bc542c7b-c42e-4e5d-bb02-b32e78c49115 nodeName:}" failed. No retries permitted until 2026-11-02 20:24:16.206092532 +0000 UTC m=+1945.080363414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls") pod "maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" (UID: "bc542c7b-c42e-4e5d-bb02-b32e78c49115") : secret "maas-api-e2e-shared-b-w6-40425f-serving-cert" not found
Nov 02 20:24:15.706768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.706746    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.708263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.708239    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:15.711757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.711726    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:24:15.763599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:15.763530    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn56\" (UniqueName: \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:16.100517 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:24:16.100421    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:24:16.235108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:16.235070    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:16.238263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:16.238231    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:16.466038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:16.466008    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:16.850137 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:16.850089    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc542c7b_c42e_4e5d_bb02_b32e78c49115.slice/crio-aa15e3f29bdb124a00b4462f3f1aaa43bd29bd0cb04707bbe1215294469912b6 WatchSource:0}: Error finding container aa15e3f29bdb124a00b4462f3f1aaa43bd29bd0cb04707bbe1215294469912b6: Status 404 returned error can't find the container with id aa15e3f29bdb124a00b4462f3f1aaa43bd29bd0cb04707bbe1215294469912b6
Nov 02 20:24:16.853815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:16.852884    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:24:17.302960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:17.302922    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" event={"ID":"bc542c7b-c42e-4e5d-bb02-b32e78c49115","Type":"ContainerStarted","Data":"aa15e3f29bdb124a00b4462f3f1aaa43bd29bd0cb04707bbe1215294469912b6"}
Nov 02 20:24:18.142410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.142369    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:24:18.148314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.148289    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.153961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.153927    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:24:18.154094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.153981    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:24:18.267232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.267193    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.283868    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.283950    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgrl\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.283995    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284068    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284122    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284210    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284270    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284338    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.284809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.284364    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.319898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.319766    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" event={"ID":"bc542c7b-c42e-4e5d-bb02-b32e78c49115","Type":"ContainerStarted","Data":"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"}
Nov 02 20:24:18.320289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.320256    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:18.386208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386168    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386242    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgrl\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386283    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386348    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386398    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386480    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386552    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386605    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.386723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386628    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.387018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.386900    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.387521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.387378    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.387521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.387449    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.387970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.387912    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.388075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.387963    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.389766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.389726    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.389879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.389854    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.432004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.431909    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" podStartSLOduration=2.618624143 podStartE2EDuration="3.431893137s" totalImagesPullingTime="813.268994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:18.429028685 +0000 UTC m=+1947.303299576" watchObservedRunningTime="2026-11-02 20:24:18.431893137 +0000 UTC m=+1947.306164025"
Nov 02 20:24:18.441456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.441426    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.442749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.442721    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgrl\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.469205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.469165    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:18.870482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.870431    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:24:18.870983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.870958    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:24:18.958274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.958221    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:18.958440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.958386    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:18.958505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.958486    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:18.962848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:18.962821    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:24:19.330606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:19.330501    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" event={"ID":"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a","Type":"ContainerStarted","Data":"2a99b5cffcb65874f60c17b2b88b386bb1ee72bc24d076fe0a8c8b382a9f9746"}
Nov 02 20:24:19.330606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:19.330571    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" event={"ID":"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a","Type":"ContainerStarted","Data":"1561f1079a6911afed3ed3819b1d8f6b5d07a33cf787fe5f4871cb1f4d6d5740"}
Nov 02 20:24:19.443006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:19.442942    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podStartSLOduration=1.442922053 podStartE2EDuration="1.442922053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:19.440604502 +0000 UTC m=+1948.314875413" watchObservedRunningTime="2026-11-02 20:24:19.442922053 +0000 UTC m=+1948.317192942"
Nov 02 20:24:19.469764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:19.469721    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:20.283868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.283070    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": read tcp 10.132.0.2:44770->10.132.0.182:15021: read: connection reset by peer" start-of-body=
Nov 02 20:24:20.283868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.283142    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": read tcp 10.132.0.2:44770->10.132.0.182:15021: read: connection reset by peer"
Nov 02 20:24:20.349415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.349379    5099 generic.go:356] "Generic (PLEG): container finished" podID="019e565a-a379-4576-b830-48d8f0252824" containerID="206164b41ae845d734b404a112bde69dadcf1716f60ea500fea0804dfc0abadd" exitCode=0
Nov 02 20:24:20.350101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.350073    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerDied","Data":"206164b41ae845d734b404a112bde69dadcf1716f60ea500fea0804dfc0abadd"}
Nov 02 20:24:20.351150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.351128    5099 scope.go:122] "RemoveContainer" containerID="206164b41ae845d734b404a112bde69dadcf1716f60ea500fea0804dfc0abadd"
Nov 02 20:24:20.354077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.354045    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.354324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.354304    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.354466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.354449    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.377803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.377762    5099 generic.go:356] "Generic (PLEG): container finished" podID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerID="9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc" exitCode=0
Nov 02 20:24:20.377942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.377844    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerDied","Data":"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"}
Nov 02 20:24:20.378579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.378550    5099 scope.go:122] "RemoveContainer" containerID="9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"
Nov 02 20:24:20.384056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.380212    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.384056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.380283    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.384056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.380322    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:20.472911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.472856    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 20:24:20.473241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:20.473166    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 20:24:21.392137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.392100    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerStarted","Data":"30f786f8c99cca94cf5e32e355a95761226b12760d7a394da627e7f7bfc6b986"}
Nov 02 20:24:21.394837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.394777    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerStarted","Data":"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"}
Nov 02 20:24:21.470845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.470406    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 20:24:21.470845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.470480    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 20:24:21.573505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.573465    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:24:21.573724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.573696    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:21.573831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.573757    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:21.963805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.963651    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:24:21.964100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.963815    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:21.964100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:21.963864    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:22.478836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.471406    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 20:24:22.478836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.471496    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 20:24:22.586971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.586927    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:22.587277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.587242    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:22.968272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.968232    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:22.968619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:22.968578    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:23.478860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.476652    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 20:24:23.487959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.487897    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 20:24:23.583120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.583081    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:23.583394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.583366    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:23.979602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.979563    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:23.979912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:23.979881    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:24.363806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.360991    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:24:24.473809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.471944    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 20:24:24.473809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.472005    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 20:24:24.592839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.577275    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:24.592839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.577343    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:24.980240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.977272    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:24.980240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:24.977352    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:25.622422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.621999    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:25.622422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.622053    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:25.625805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.623103    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:25.625805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.624205    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:25.625805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.625255    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:24:25.757366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.757307    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:24:25.959510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.959403    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\") pod \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") "
Nov 02 20:24:25.959702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.959629    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") pod \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") "
Nov 02 20:24:25.992579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:25.983065    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume" pod "b133cdf1-88c5-4a41-8619-10ef0aaa874b" (UID: "b133cdf1-88c5-4a41-8619-10ef0aaa874b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:24:26.017808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.013230    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:26.017808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.013289    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:26.064508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.063994    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn" pod "b133cdf1-88c5-4a41-8619-10ef0aaa874b" (UID: "b133cdf1-88c5-4a41-8619-10ef0aaa874b"). InnerVolumeSpecName "kube-api-access-bx9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:24:26.079989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.079950    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") pod \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\" (UID: \"b133cdf1-88c5-4a41-8619-10ef0aaa874b\") "
Nov 02 20:24:26.089095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.089059    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b133cdf1-88c5-4a41-8619-10ef0aaa874b-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:26.092999 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:26.092962    5099 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/b133cdf1-88c5-4a41-8619-10ef0aaa874b/volumes/kubernetes.io~projected/kube-api-access-bx9dn
Nov 02 20:24:26.093246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.093210    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn" pod "b133cdf1-88c5-4a41-8619-10ef0aaa874b" (UID: "b133cdf1-88c5-4a41-8619-10ef0aaa874b"). InnerVolumeSpecName "kube-api-access-bx9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:24:26.119736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.119501    5099 generic.go:356] "Generic (PLEG): container finished" podID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerID="e94aa3f8f8f9f2f10d393b40176bc7f77f09e52e9259b534eb1507a93eb3d075" exitCode=0
Nov 02 20:24:26.119736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.119675    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" event={"ID":"6e00f63b-f05c-4e01-b83b-5753096a40cb","Type":"ContainerDied","Data":"e94aa3f8f8f9f2f10d393b40176bc7f77f09e52e9259b534eb1507a93eb3d075"}
Nov 02 20:24:26.122243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.122207    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"
Nov 02 20:24:26.122370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.122239    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" event={"ID":"b133cdf1-88c5-4a41-8619-10ef0aaa874b","Type":"ContainerDied","Data":"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"}
Nov 02 20:24:26.122370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.122309    5099 scope.go:122] "RemoveContainer" containerID="7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"
Nov 02 20:24:26.125681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.125653    5099 generic.go:356] "Generic (PLEG): container finished" podID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerID="7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f" exitCode=0
Nov 02 20:24:26.125813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.125768    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4" event={"ID":"b133cdf1-88c5-4a41-8619-10ef0aaa874b","Type":"ContainerDied","Data":"78486fa38c0ee31c72b60355d354e1869cea347f5bf8e0ed8bfe0955f565ac35"}
Nov 02 20:24:26.226982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.226947    5099 scope.go:122] "RemoveContainer" containerID="7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"
Nov 02 20:24:26.229033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.228993    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9dn\" (UniqueName: \"kubernetes.io/projected/b133cdf1-88c5-4a41-8619-10ef0aaa874b-kube-api-access-bx9dn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:26.271820 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:24:26.270468    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f\": container with ID starting with 7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f not found: ID does not exist" containerID="7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"
Nov 02 20:24:26.271820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.270510    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f"} err="failed to get container status \"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f\": rpc error: code = NotFound desc = could not find container \"7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f\": container with ID starting with 7a27012be95861dc5a30fad39a74e09ed8d4702d1a126bed85799a7cbffd6f8f not found: ID does not exist"
Nov 02 20:24:26.325286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.325193    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:24:26.421175 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.421138    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:24:26.425430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.425407    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-lzqd4"]
Nov 02 20:24:26.431395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.431371    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\" (UniqueName: \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\") pod \"6e00f63b-f05c-4e01-b83b-5753096a40cb\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") "
Nov 02 20:24:26.431505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.431455    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\") pod \"6e00f63b-f05c-4e01-b83b-5753096a40cb\" (UID: \"6e00f63b-f05c-4e01-b83b-5753096a40cb\") "
Nov 02 20:24:26.431833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.431811    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume" pod "6e00f63b-f05c-4e01-b83b-5753096a40cb" (UID: "6e00f63b-f05c-4e01-b83b-5753096a40cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:24:26.432125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.432110    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e00f63b-f05c-4e01-b83b-5753096a40cb-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:26.433584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.433565    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb" pod "6e00f63b-f05c-4e01-b83b-5753096a40cb" (UID: "6e00f63b-f05c-4e01-b83b-5753096a40cb"). InnerVolumeSpecName "kube-api-access-7vvxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:24:26.533596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.533514    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvxb\" (UniqueName: \"kubernetes.io/projected/6e00f63b-f05c-4e01-b83b-5753096a40cb-kube-api-access-7vvxb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:24:26.575983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.575945    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:26.576757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.576726    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:26.966806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.964388    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 02 20:24:26.966806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:26.964448    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 02 20:24:27.160807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.152842    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"
Nov 02 20:24:27.160807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.155304    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g" event={"ID":"6e00f63b-f05c-4e01-b83b-5753096a40cb","Type":"ContainerDied","Data":"2d43b1ca64af915d55a784d703ef0fdd9b1e70659233473eabc37d9fbe93e81e"}
Nov 02 20:24:27.160807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.155400    5099 scope.go:122] "RemoveContainer" containerID="e94aa3f8f8f9f2f10d393b40176bc7f77f09e52e9259b534eb1507a93eb3d075"
Nov 02 20:24:27.313318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.313241    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" path="/var/lib/kubelet/pods/b133cdf1-88c5-4a41-8619-10ef0aaa874b/volumes"
Nov 02 20:24:27.395807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.394845    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:24:27.402804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.401170    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-8h77g"]
Nov 02 20:24:27.583864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.583221    5099 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 20:24:27.583864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.583279    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 20:24:27.974261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.974230    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:24:27.974718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.974700    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:24:27.975584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:27.975562    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:24:28.577389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:28.577357    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:24:28.577848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:28.577820    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:24:28.578735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:28.578715    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:24:29.263655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:29.263617    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" path="/var/lib/kubelet/pods/6e00f63b-f05c-4e01-b83b-5753096a40cb/volumes"
Nov 02 20:24:45.732467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.732431    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:24:45.733309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733289    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing"
Nov 02 20:24:45.733309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733309    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing"
Nov 02 20:24:45.733445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733331    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing"
Nov 02 20:24:45.733445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733337    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing"
Nov 02 20:24:45.733511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733492    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b133cdf1-88c5-4a41-8619-10ef0aaa874b" containerName="payload-processing"
Nov 02 20:24:45.733511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.733502    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e00f63b-f05c-4e01-b83b-5753096a40cb" containerName="payload-pre-processing"
Nov 02 20:24:45.737442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.737424    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.779945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.779909    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:24:45.784652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.784635    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.828850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.828816    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:24:45.842844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.842805    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgh4g\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.842920    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843000    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843043    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843098    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843176    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843213    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843334    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.843406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.843385    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.884867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.884827    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:24:45.944571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944537    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.944765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944607    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm4f\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.944765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944670    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.944765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944703    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.944765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944726    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.944765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944750    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.945172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.944848    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.945326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.945289    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.945420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.945400    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.945509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.945434    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.945509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.945464    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.945835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.945811    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.946797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.946144    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgh4g\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.946915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.946851    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.949057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.949004    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.950123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.950100    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.950870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.950849    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.951488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.951269    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.954497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.954457    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.954678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.954644    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.954860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.954838    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:45.955569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.955526    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.955690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.955672    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.956132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.956110    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.962309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.962283    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.999135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.999069    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:45.999674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:45.999654    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgh4g\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:46.054325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.054293    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:46.056988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.056955    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057000    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057042    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057085    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057177    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057237    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057294    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm4f\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057355    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057392    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057422    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057442    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057540    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.057705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.057636    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.058348 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.058306    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.059452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.059425    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.059688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.059670    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.097904 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:24:46.097855    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:24:46.111183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.111150    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.111346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.111283    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm4f\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.399721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.399686    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:46.496933 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:46.496901    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce869ac9_9cef_4476_ad89_0def89d9fe65.slice/crio-9c4396a8398fb1b3cee6d707dbd0befb5838625a5a2b4a0512e58c8e8836f0b4 WatchSource:0}: Error finding container 9c4396a8398fb1b3cee6d707dbd0befb5838625a5a2b4a0512e58c8e8836f0b4: Status 404 returned error can't find the container with id 9c4396a8398fb1b3cee6d707dbd0befb5838625a5a2b4a0512e58c8e8836f0b4
Nov 02 20:24:46.499097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.499056    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:24:46.500116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.500072    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:46.500210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.500162    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:46.500276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.500207    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:46.782658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.782561    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:24:46.786958 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:46.786929    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8017ac0c_b3d3_4318_b821_200341fb27ef.slice/crio-8574b12adc2d076e19748ee5e0a48af9c8082b0c50d930310433e90125fc3ff5 WatchSource:0}: Error finding container 8574b12adc2d076e19748ee5e0a48af9c8082b0c50d930310433e90125fc3ff5: Status 404 returned error can't find the container with id 8574b12adc2d076e19748ee5e0a48af9c8082b0c50d930310433e90125fc3ff5
Nov 02 20:24:46.791510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.791470    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:46.791592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.791565    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:46.791655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:46.791609    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:47.392818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.392363    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" event={"ID":"ce869ac9-9cef-4476-ad89-0def89d9fe65","Type":"ContainerStarted","Data":"0997067a5f2e174d4a56ad514e35790cec8e4fed06bb7155f90b066590e8b51d"}
Nov 02 20:24:47.392818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.392434    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" event={"ID":"ce869ac9-9cef-4476-ad89-0def89d9fe65","Type":"ContainerStarted","Data":"9c4396a8398fb1b3cee6d707dbd0befb5838625a5a2b4a0512e58c8e8836f0b4"}
Nov 02 20:24:47.396849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.396014    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" event={"ID":"8017ac0c-b3d3-4318-b821-200341fb27ef","Type":"ContainerStarted","Data":"38ae8d936088a1165f6637433e8f3072c6413bd3870f8f0b5eea4ea645dcbe1a"}
Nov 02 20:24:47.396849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.396076    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" event={"ID":"8017ac0c-b3d3-4318-b821-200341fb27ef","Type":"ContainerStarted","Data":"8574b12adc2d076e19748ee5e0a48af9c8082b0c50d930310433e90125fc3ff5"}
Nov 02 20:24:47.400087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.400067    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:47.402440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.402407    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:47.402594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.402576    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:47.510758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.510699    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podStartSLOduration=2.5106805530000003 podStartE2EDuration="2.510680553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:47.505611425 +0000 UTC m=+1976.379882324" watchObservedRunningTime="2026-11-02 20:24:47.510680553 +0000 UTC m=+1976.384951442"
Nov 02 20:24:47.619284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:47.619222    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podStartSLOduration=2.6192065639999997 podStartE2EDuration="2.619206564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:47.615557183 +0000 UTC m=+1976.489828085" watchObservedRunningTime="2026-11-02 20:24:47.619206564 +0000 UTC m=+1976.493477453"
Nov 02 20:24:48.058808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:48.058190    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:48.063803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:48.062909    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:48.063803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:48.062989    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:48.415804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:48.403198    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:48.415804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:48.403257    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:49.059942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:49.059907    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:49.060656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:49.060622    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:49.400184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:49.400146    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:49.400407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:49.400259    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:50.058808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:50.055295    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:50.058808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:50.055362    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:50.401032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:50.400933    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:50.401032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:50.400989    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:51.057973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:51.057932    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:51.058262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:51.058232    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:51.402181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:51.401120    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:51.402181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:51.401201    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:52.055217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:52.055175    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:52.055523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:52.055491    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:52.412807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:52.401198    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:52.412807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:52.401260    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:53.074810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.063234    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 20:24:53.074810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.063298    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 20:24:53.247807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.237346    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:24:53.251141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.251117    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.285925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.285881    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.285925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.285923    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286033    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286083    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stc7d\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286233    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286351    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286411    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286481    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.286557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.286543    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.300748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.300717    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-81e58f-openshift-default-dockercfg-gxnmk\""
Nov 02 20:24:53.388129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.388093    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.388378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.388360    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.389581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.389551    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.389826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.389808    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stc7d\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.390015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.390001    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.390184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.390170    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.390296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.390283    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.390443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.390427    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.390576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.390563    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.391178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.391155    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.391584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.391565    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.392262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.392243    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.393057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.392449    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.393184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.392997    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.400233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.400211    5099 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 20:24:53.401889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.401861    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 20:24:53.402385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.400956    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.420441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.420363    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.422432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.422320    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:24:53.663114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.663033    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stc7d\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.838062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.837192    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\") pod \"e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") " pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:53.875075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:53.875026    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:54.060075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.060040    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:54.060593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.060573    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:54.061422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.061397    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:24:54.167700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.167663    5099 scope.go:122] "RemoveContainer" containerID="49dc56a65acf8d341e00f39403037b59616046e713bf8a78b3420ec135f66f19"
Nov 02 20:24:54.324270 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:54.324233    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode793033c_8ad8_4d36_87cc_0de4a0a7eda5.slice/crio-4176a0b8e445225e7bf879a8194098a6e83ad2f6defdd4183d3d03a216174ecb WatchSource:0}: Error finding container 4176a0b8e445225e7bf879a8194098a6e83ad2f6defdd4183d3d03a216174ecb: Status 404 returned error can't find the container with id 4176a0b8e445225e7bf879a8194098a6e83ad2f6defdd4183d3d03a216174ecb
Nov 02 20:24:54.325515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.325489    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:24:54.327310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.327278    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:54.327391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.327340    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:54.327391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.327370    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:54.412261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.412229    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:54.412830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.412765    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:54.413956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.413885    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:24:54.860828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.858904    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerStarted","Data":"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"}
Nov 02 20:24:54.860828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.858968    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerStarted","Data":"4176a0b8e445225e7bf879a8194098a6e83ad2f6defdd4183d3d03a216174ecb"}
Nov 02 20:24:54.875662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.875632    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:54.970447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:54.970393    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podStartSLOduration=1.970378774 podStartE2EDuration="1.970378774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:54.969342027 +0000 UTC m=+1983.843612913" watchObservedRunningTime="2026-11-02 20:24:54.970378774 +0000 UTC m=+1983.844649719"
Nov 02 20:24:55.882632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:55.882596    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:56.877710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:56.877681    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:56.878736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:56.878713    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:24:58.085298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.085250    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:24:58.102003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.101965    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.177523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.177482    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-429b45-openshift-default-dockercfg-htmgg\""
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.362983    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363034    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363084    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363165    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363234    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363320    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363438    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363596    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.370822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.363769    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7plf\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.548268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.548233    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:24:58.584018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.583981    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7plf\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584279    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584403    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584503    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584625    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584756 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584742    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.584929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.584910    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.585101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.585086    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.585293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.585278    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.585913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.585887    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.586933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.586910    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.587313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.587292    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.618931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.618468    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.619551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.619496    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.649913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.644339    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.668304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.668220    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.694817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.691568    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7plf\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.694817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.694413    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\") pod \"e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") " pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:58.773806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:58.773100    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:24:59.235395 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:24:59.235366    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f467d9_f03b_41db_8ca3_bb9729a2b6d9.slice/crio-a7008573124d888399641857e1b5d7ef4ff0b4ba32cc877fa8c807ed229df0e4 WatchSource:0}: Error finding container a7008573124d888399641857e1b5d7ef4ff0b4ba32cc877fa8c807ed229df0e4: Status 404 returned error can't find the container with id a7008573124d888399641857e1b5d7ef4ff0b4ba32cc877fa8c807ed229df0e4
Nov 02 20:24:59.241676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.237307    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:24:59.244433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.244396    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:59.244528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.244471    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:59.244528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.244501    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:24:59.911016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.910976    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerStarted","Data":"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"}
Nov 02 20:24:59.911227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:24:59.911029    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerStarted","Data":"a7008573124d888399641857e1b5d7ef4ff0b4ba32cc877fa8c807ed229df0e4"}
Nov 02 20:25:00.072191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:00.072133    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podStartSLOduration=2.072117003 podStartE2EDuration="2.072117003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:00.067676082 +0000 UTC m=+1988.941946970" watchObservedRunningTime="2026-11-02 20:25:00.072117003 +0000 UTC m=+1988.946387890"
Nov 02 20:25:00.774167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:00.774128    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:25:00.780763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:00.780733    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:25:00.919041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:00.919014    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:25:00.919892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:00.919871    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:25:02.437658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.437618    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:25:02.815264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.815177    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:25:02.815588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.815565    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:02.887801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.868293    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-81e58f\""
Nov 02 20:25:02.896986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.889822    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp77\" (UniqueName: \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:02.902637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.902586    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:02.927214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:02.927187    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-81e58f-dockercfg-lts5q\""
Nov 02 20:25:03.004566 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.004536    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp77\" (UniqueName: \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:03.004870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.004854    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:03.005706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.005686    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:03.061836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.060965    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp77\" (UniqueName: \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\") pod \"payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:03.248481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.248446    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:03.677918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:03.657852    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:25:03.677918 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:25:03.676396    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aab373a_5fe7_4192_b54c_2de3093b0284.slice/crio-2a29366edfc3a2ecf4f67828f39fb14dbd610c9e415abb6917204d42f4da6d44 WatchSource:0}: Error finding container 2a29366edfc3a2ecf4f67828f39fb14dbd610c9e415abb6917204d42f4da6d44: Status 404 returned error can't find the container with id 2a29366edfc3a2ecf4f67828f39fb14dbd610c9e415abb6917204d42f4da6d44
Nov 02 20:25:04.033088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:04.032996    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" event={"ID":"4aab373a-5fe7-4192-b54c-2de3093b0284","Type":"ContainerStarted","Data":"7e3f79fe3ca4e99392eade93df3c07678e160874f91cbca5a93b60f4f5964e8c"}
Nov 02 20:25:04.033088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:04.033064    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:04.033088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:04.033086    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" event={"ID":"4aab373a-5fe7-4192-b54c-2de3093b0284","Type":"ContainerStarted","Data":"2a29366edfc3a2ecf4f67828f39fb14dbd610c9e415abb6917204d42f4da6d44"}
Nov 02 20:25:04.149432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:04.149365    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" podStartSLOduration=2.149343323 podStartE2EDuration="2.149343323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:04.143831461 +0000 UTC m=+1993.018102364" watchObservedRunningTime="2026-11-02 20:25:04.149343323 +0000 UTC m=+1993.023614212"
Nov 02 20:25:15.045177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:15.045145    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:25:15.078981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:15.078935    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:25:15.079148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:15.079046    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:25:16.076568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.076534    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:25:16.097450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.097421    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:25:16.097773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.097713    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.132168 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:25:16.132109    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:25:16.203855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.203761    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-429b45\""
Nov 02 20:25:16.203855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.203777    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-429b45-dockercfg-qz9k2\""
Nov 02 20:25:16.285946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.285864    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.285946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.285919    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4nm\" (UniqueName: \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.388630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.388574    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.388877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.388640    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4nm\" (UniqueName: \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.390148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.390115    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.460973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.460897    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4nm\" (UniqueName: \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\") pod \"payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") " pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:16.720953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:16.720857    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:17.107281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:17.107245    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:25:17.109941 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:25:17.109912    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df451b8_1cfd_4b23_b289_30257a582317.slice/crio-05fe958e27a4804a6426c354c640b0e2217c189e246a5dfc234e84856c36ad41 WatchSource:0}: Error finding container 05fe958e27a4804a6426c354c640b0e2217c189e246a5dfc234e84856c36ad41: Status 404 returned error can't find the container with id 05fe958e27a4804a6426c354c640b0e2217c189e246a5dfc234e84856c36ad41
Nov 02 20:25:17.962960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:17.962923    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" event={"ID":"6df451b8-1cfd-4b23-b289-30257a582317","Type":"ContainerStarted","Data":"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"}
Nov 02 20:25:17.963167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:17.962979    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" event={"ID":"6df451b8-1cfd-4b23-b289-30257a582317","Type":"ContainerStarted","Data":"05fe958e27a4804a6426c354c640b0e2217c189e246a5dfc234e84856c36ad41"}
Nov 02 20:25:17.963167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:17.963085    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:18.099277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.099200    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" podStartSLOduration=3.099163004 podStartE2EDuration="3.099163004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:18.080223545 +0000 UTC m=+2006.954494424" watchObservedRunningTime="2026-11-02 20:25:18.099163004 +0000 UTC m=+2006.973433895"
Nov 02 20:25:18.316927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.316845    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:25:18.335981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.335802    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.419005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.417017    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8c5e8e\""
Nov 02 20:25:18.421482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.421357    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:25:18.463844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.463124    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7njs\" (UniqueName: \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.463844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.463425    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.479814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.478245    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:25:18.764091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.743877    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.764091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.744115    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7njs\" (UniqueName: \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.776877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.765187    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.795546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.795405    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:18.813712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.813523    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:25:18.821512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.821486    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7njs\" (UniqueName: \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\") pod \"payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:18.847388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.847357    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrfw\" (UniqueName: \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:18.847513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.847460    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:18.861182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.861148    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8c5e8e-dockercfg-dkc76\""
Nov 02 20:25:18.948847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.948807    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrfw\" (UniqueName: \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:18.949041 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.948952    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:18.950138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:18.950104    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:19.002687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.002655    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrfw\" (UniqueName: \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\") pod \"payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") " pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:19.035428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.035346    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:19.156831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.156775    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:19.419059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.419030    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:25:19.420055 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:25:19.420016    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eab544_7657_4c12_a9da_7cc658a0bfa9.slice/crio-63b2480b988e4916b8a41c11b322a32365e2d949a3c9598049cc32c00c3bc26e WatchSource:0}: Error finding container 63b2480b988e4916b8a41c11b322a32365e2d949a3c9598049cc32c00c3bc26e: Status 404 returned error can't find the container with id 63b2480b988e4916b8a41c11b322a32365e2d949a3c9598049cc32c00c3bc26e
Nov 02 20:25:19.555603 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:25:19.555575    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66604f72_a49b_41aa_ba70_26dc77341d00.slice/crio-56cc9dfb23e41f6189841de46df4769fa640780b86e6d74f0bf254abc3302720 WatchSource:0}: Error finding container 56cc9dfb23e41f6189841de46df4769fa640780b86e6d74f0bf254abc3302720: Status 404 returned error can't find the container with id 56cc9dfb23e41f6189841de46df4769fa640780b86e6d74f0bf254abc3302720
Nov 02 20:25:19.573246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.573213    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:25:19.998612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.998531    5099 generic.go:356] "Generic (PLEG): container finished" podID="982ee009-cc01-4529-bd61-abbefa04fec6" containerID="7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea" exitCode=0
Nov 02 20:25:19.998903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.998621    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerDied","Data":"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"}
Nov 02 20:25:19.999451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:19.999428    5099 scope.go:122] "RemoveContainer" containerID="7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"
Nov 02 20:25:20.000740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.000718    5099 generic.go:356] "Generic (PLEG): container finished" podID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerID="8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3" exitCode=0
Nov 02 20:25:20.001424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.000819    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerDied","Data":"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"}
Nov 02 20:25:20.001424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.001362    5099 scope.go:122] "RemoveContainer" containerID="8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"
Nov 02 20:25:20.001834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.001797    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.001959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.001884    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.001959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.001929    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.002901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.002874    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" event={"ID":"c3eab544-7657-4c12-a9da-7cc658a0bfa9","Type":"ContainerStarted","Data":"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"}
Nov 02 20:25:20.003021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.002935    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" event={"ID":"c3eab544-7657-4c12-a9da-7cc658a0bfa9","Type":"ContainerStarted","Data":"63b2480b988e4916b8a41c11b322a32365e2d949a3c9598049cc32c00c3bc26e"}
Nov 02 20:25:20.003021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.002970    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:20.003332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.003306    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.003435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.003382    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.003501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.003435    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:20.005434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.005411    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" event={"ID":"66604f72-a49b-41aa-ba70-26dc77341d00","Type":"ContainerStarted","Data":"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"}
Nov 02 20:25:20.005545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.005457    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" event={"ID":"66604f72-a49b-41aa-ba70-26dc77341d00","Type":"ContainerStarted","Data":"56cc9dfb23e41f6189841de46df4769fa640780b86e6d74f0bf254abc3302720"}
Nov 02 20:25:20.005545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.005482    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:20.244637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.244520    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" podStartSLOduration=2.24449894 podStartE2EDuration="2.24449894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:20.224485108 +0000 UTC m=+2009.098755997" watchObservedRunningTime="2026-11-02 20:25:20.24449894 +0000 UTC m=+2009.118769828"
Nov 02 20:25:20.351741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:20.351617    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" podStartSLOduration=2.351598405 podStartE2EDuration="2.351598405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:20.333481203 +0000 UTC m=+2009.207752094" watchObservedRunningTime="2026-11-02 20:25:20.351598405 +0000 UTC m=+2009.225869293"
Nov 02 20:25:21.017435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.017399    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerStarted","Data":"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"}
Nov 02 20:25:21.019853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.019827    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerStarted","Data":"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"}
Nov 02 20:25:21.221511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.221466    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:25:21.222210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.222166    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:21.222350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.222227    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:21.926214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.926176    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:25:21.927693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.927668    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 20:25:21.927759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:21.927722    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 20:25:22.222510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:22.222420    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:22.222510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:22.222484    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:22.926561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:22.926521    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 20:25:22.926740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:22.926578    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 20:25:23.221476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:23.221391    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:23.221476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:23.221463    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:23.926562 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:23.926523    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 20:25:23.926998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:23.926585    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 20:25:24.221661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.221571    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:24.221953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.221924    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:24.926689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.926650    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 20:25:24.927104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.926705    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 20:25:24.928397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.928359    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:25:24.932969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:24.932953    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.013804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.013705    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d5a51f-openshift-default-dockercfg-6hv7q\""
Nov 02 20:25:25.028618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.028572    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.032598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.032567    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.032896    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8rl\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.032964    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.033091    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.033254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.033358    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.033404    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.033695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.033451    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.151632    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162115    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162227    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162271    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162315    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162388    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162421    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162495    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8rl\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162538    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.162656    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.178913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.163229    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.201891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.180078    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.201891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.180351    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.201891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.180611    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.201891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.181836    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.231892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.223241    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:25.231892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.223364    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:25.413830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.412031    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8rl\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.535853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.532524    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.549855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.536761    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.591159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.542668    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\") pod \"e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") " pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:25.621119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:25.621082    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:26.040418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.038910    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 02 20:25:26.040418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.038966    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 02 20:25:26.223703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.223662    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 20:25:26.223931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.223718    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 20:25:26.408283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.407567    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:25:26.410358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.410316    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:26.410559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.410526    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:26.410745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.410711    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:26.638837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.636553    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerStarted","Data":"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"}
Nov 02 20:25:26.638837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.636602    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerStarted","Data":"acd87c4ce9e4e5335ad0b83835843d013608f7cd26a3533e46759f9a4a857578"}
Nov 02 20:25:26.941808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.941202    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:25:26.960728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.942871    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:25:26.960728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:26.944837    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:25:27.061000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.060950    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podStartSLOduration=3.060934103 podStartE2EDuration="3.060934103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:26.748014025 +0000 UTC m=+2015.622284915" watchObservedRunningTime="2026-11-02 20:25:27.060934103 +0000 UTC m=+2015.935204991"
Nov 02 20:25:27.226287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.226198    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:25:27.226596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.226574    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:25:27.227161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.227128    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:25:27.622623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.622592    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:27.628715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.628691    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:27.645960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.645934    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:27.646442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:27.646421    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:25:29.009824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:29.007773    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:25:31.026967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:31.026753    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:25:31.030461 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:31.030434    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:25:38.040483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:38.039289    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:25:38.040483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:38.039631    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" containerID="cri-o://aca6070f3e74b6179847a0d886af75cf6b83ba4c4e080f31ad407e7a42946b61" gracePeriod=30
Nov 02 20:25:40.286899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:40.286857    5099 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:25:40.287290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:40.286918    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:25:44.173803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.173308    5099 generic.go:356] "Generic (PLEG): container finished" podID="2095e6d0-8120-4baa-82d0-79685a165272" containerID="aca6070f3e74b6179847a0d886af75cf6b83ba4c4e080f31ad407e7a42946b61" exitCode=0
Nov 02 20:25:44.173803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.173412    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" event={"ID":"2095e6d0-8120-4baa-82d0-79685a165272","Type":"ContainerDied","Data":"aca6070f3e74b6179847a0d886af75cf6b83ba4c4e080f31ad407e7a42946b61"}
Nov 02 20:25:44.326999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.326972    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:25:44.459161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459072    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459202    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459244    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459317    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459341    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459342    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:44.459497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459415    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.459497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459477    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.460142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459558    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.460142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459619    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:44.460142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459645    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\") pod \"2095e6d0-8120-4baa-82d0-79685a165272\" (UID: \"2095e6d0-8120-4baa-82d0-79685a165272\") "
Nov 02 20:25:44.460142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459828    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:44.460142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.459853    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:44.460465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460264    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:25:44.460560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460542    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.460622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460561    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.460622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460571    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2095e6d0-8120-4baa-82d0-79685a165272-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.460622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460580    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.460622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.460589    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.461908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.461879    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:25:44.462025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.461956    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "kube-api-access-m6c7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:25:44.462025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.461999    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:25:44.462400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.462379    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy" pod "2095e6d0-8120-4baa-82d0-79685a165272" (UID: "2095e6d0-8120-4baa-82d0-79685a165272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:25:44.561846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.561807    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6c7k\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-kube-api-access-m6c7k\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.561846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.561838    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2095e6d0-8120-4baa-82d0-79685a165272-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.561846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.561856    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2095e6d0-8120-4baa-82d0-79685a165272-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.562111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.561866    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2095e6d0-8120-4baa-82d0-79685a165272-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:25:44.688810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.684100    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"]
Nov 02 20:25:44.704449 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:44.704209    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-d66bp"]
Nov 02 20:25:45.270056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.270029    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"
Nov 02 20:25:45.276892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.276858    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5195e7d6-2f83-450f-9d96-5b88bd22b02c" path="/var/lib/kubelet/pods/5195e7d6-2f83-450f-9d96-5b88bd22b02c/volumes"
Nov 02 20:25:45.277711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.277688    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg" event={"ID":"2095e6d0-8120-4baa-82d0-79685a165272","Type":"ContainerDied","Data":"88ad431950ef0c6e19304e0226206276a06e6b17b721c418b47d14ca7ff7f94a"}
Nov 02 20:25:45.281418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.277750    5099 scope.go:122] "RemoveContainer" containerID="aca6070f3e74b6179847a0d886af75cf6b83ba4c4e080f31ad407e7a42946b61"
Nov 02 20:25:45.433217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.433179    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:25:45.473049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.473009    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-l7qvg"]
Nov 02 20:25:45.812446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.812406    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:25:45.813697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.813676    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy"
Nov 02 20:25:45.813697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.813697    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy"
Nov 02 20:25:45.813992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.813979    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2095e6d0-8120-4baa-82d0-79685a165272" containerName="istio-proxy"
Nov 02 20:25:45.819597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.819578    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.869082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.869048    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-abc937-openshift-default-dockercfg-7d6xk\""
Nov 02 20:25:45.871215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.871182    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:25:45.907668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.907632    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkkz\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.907867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.907674    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.907867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.907701    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.907911    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.908024    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.908149    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908281 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.908254    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.908325    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:45.908645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:45.908455    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.012737    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013223    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013445    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013601    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013669    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013709    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013720    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013891    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.013970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013943    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkkz\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.014178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.013972    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.014178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.014008    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.014625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.014580    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.015873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.014936    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.015873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.015307    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.018120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.018098    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.073395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.066228    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.086619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.086055    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.094142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.093972    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkkz\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\") pod \"e2e-api-a-abc937-openshift-default-8884f867f-stldm\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") " pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.122081 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:25:46.122043    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:25:46.135023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.134996    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:46.562336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.562304    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:25:46.598050 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:25:46.598018    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864dc561_84ac_4b6e_9151_7fe3b0f21f2e.slice/crio-bb7dde157a985f748215d183583effc694fd6d857acf7ee12136f79faca788dd WatchSource:0}: Error finding container bb7dde157a985f748215d183583effc694fd6d857acf7ee12136f79faca788dd: Status 404 returned error can't find the container with id bb7dde157a985f748215d183583effc694fd6d857acf7ee12136f79faca788dd
Nov 02 20:25:46.601172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.601138    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:46.601284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.601222    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:46.601284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:46.601270    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:25:47.261705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:47.261668    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2095e6d0-8120-4baa-82d0-79685a165272" path="/var/lib/kubelet/pods/2095e6d0-8120-4baa-82d0-79685a165272/volumes"
Nov 02 20:25:47.290589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:47.290549    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerStarted","Data":"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"}
Nov 02 20:25:47.290804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:47.290619    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerStarted","Data":"bb7dde157a985f748215d183583effc694fd6d857acf7ee12136f79faca788dd"}
Nov 02 20:25:47.405190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:47.405141    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podStartSLOduration=2.405124564 podStartE2EDuration="2.405124564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:47.401196793 +0000 UTC m=+2036.275467681" watchObservedRunningTime="2026-11-02 20:25:47.405124564 +0000 UTC m=+2036.279395455"
Nov 02 20:25:48.135559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:48.135518    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:48.141734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:48.141708    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:48.298898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:48.298853    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:48.300169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:48.300142    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:25:54.524806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:54.499187    5099 scope.go:122] "RemoveContainer" containerID="d450539bd95cb6e458fcf6e7e3a3ea322905deb0a0c1f27353691d4498cc6b81"
Nov 02 20:25:54.899206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:54.899179    5099 scope.go:122] "RemoveContainer" containerID="ad281fd0b55213ae5da4c8fa17491c564420e9f6f5c3936d0aa0ca8b9a7d8da0"
Nov 02 20:25:54.949837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:25:54.949589    5099 scope.go:122] "RemoveContainer" containerID="84d9b9b5c47a16505852b8a06f0c7545ba5b53da0149a64e552a87737b6f7b71"
Nov 02 20:26:08.262893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:08.262422    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:26:08.262893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:08.262653    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy" containerID="cri-o://18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c" gracePeriod=30
Nov 02 20:26:13.565686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.565654    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:26:13.614909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.614874    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.614934    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615075    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615111    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615134    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:26:13.615247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615199    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615263    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615332    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:26:13.615460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615438    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615480    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615499    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:26:13.615605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615586    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\") pod \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\" (UID: \"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64\") "
Nov 02 20:26:13.615664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615641    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:26:13.615851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.615777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:26:13.616502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.616477    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.616616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.616506    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.616616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.616522    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.616616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.616535    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.616616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.616548    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.617591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.617565    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:26:13.617591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.617575    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:26:13.617726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.617606    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "kube-api-access-j4nbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:26:13.617800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.617764    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo" pod "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" (UID: "ca717e9b-8d99-4caa-bf89-fd40b1ae6a64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:26:13.718526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.718482    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.718526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.718522    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.718526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.718531    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:13.718828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:13.718540    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nbk\" (UniqueName: \"kubernetes.io/projected/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64-kube-api-access-j4nbk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:26:14.213766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.213723    5099 generic.go:356] "Generic (PLEG): container finished" podID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerID="18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c" exitCode=0
Nov 02 20:26:14.213766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.213791    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"
Nov 02 20:26:14.213766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.213802    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" event={"ID":"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64","Type":"ContainerDied","Data":"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"}
Nov 02 20:26:14.214120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.213857    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg" event={"ID":"ca717e9b-8d99-4caa-bf89-fd40b1ae6a64","Type":"ContainerDied","Data":"75719898116bab9c452578fc126e4da7a8d4a6d1058a4a5e2838423032862e61"}
Nov 02 20:26:14.214120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.213885    5099 scope.go:122] "RemoveContainer" containerID="18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"
Nov 02 20:26:14.227905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.227887    5099 scope.go:122] "RemoveContainer" containerID="18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"
Nov 02 20:26:14.228156 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:26:14.228130    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c\": container with ID starting with 18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c not found: ID does not exist" containerID="18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"
Nov 02 20:26:14.228208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.228169    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c"} err="failed to get container status \"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c\": rpc error: code = NotFound desc = could not find container \"18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c\": container with ID starting with 18d7ab41f9bf43f63b25b6f9d711215b84dff4e256c124942b18da03880c033c not found: ID does not exist"
Nov 02 20:26:14.373625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.373587    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:26:14.420504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:14.420462    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-8nsmg"]
Nov 02 20:26:15.261919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:15.261884    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" path="/var/lib/kubelet/pods/ca717e9b-8d99-4caa-bf89-fd40b1ae6a64/volumes"
Nov 02 20:26:16.096642 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:26:16.096606    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:26:40.779993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.779958    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:26:40.781165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.781144    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy"
Nov 02 20:26:40.781226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.781166    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy"
Nov 02 20:26:40.781395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.781385    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca717e9b-8d99-4caa-bf89-fd40b1ae6a64" containerName="istio-proxy"
Nov 02 20:26:40.786980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.786960    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.838365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.838328    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:26:40.878908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.878871    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8c5e8e-serving-cert\""
Nov 02 20:26:40.893678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.893638    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.893869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.893771    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s7x\" (UniqueName: \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.893959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.893909    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.894025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.893964    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.996152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.996115    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.996328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.996166    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.996385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.996353    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.996434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.996405    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s7x\" (UniqueName: \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.996556 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:26:40.996528    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-serving-cert: secret "maas-api-e2e-api-b-8c5e8e-serving-cert" not found
Nov 02 20:26:40.996654 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:26:40.996624    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls podName:fcf57063-a180-4370-8794-37e58ffbe2e7 nodeName:}" failed. No retries permitted until 2026-11-02 20:26:41.496599635 +0000 UTC m=+2090.370870501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls") pod "maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" (UID: "fcf57063-a180-4370-8794-37e58ffbe2e7") : secret "maas-api-e2e-api-b-8c5e8e-serving-cert" not found
Nov 02 20:26:40.996905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.996883    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:40.998933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:40.998902    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:41.098855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:41.096882    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s7x\" (UniqueName: \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:41.509422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:41.509384    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:41.512023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:41.511988    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") pod \"maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:41.702055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:41.702015    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:42.093642 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:26:42.093605    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf57063_a180_4370_8794_37e58ffbe2e7.slice/crio-3fe1e1de891ef3ee10c8939ac75e847b5357b9c9a82c0c25c38558831f44e0d3 WatchSource:0}: Error finding container 3fe1e1de891ef3ee10c8939ac75e847b5357b9c9a82c0c25c38558831f44e0d3: Status 404 returned error can't find the container with id 3fe1e1de891ef3ee10c8939ac75e847b5357b9c9a82c0c25c38558831f44e0d3
Nov 02 20:26:42.094499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:42.094473    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:26:42.095344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:42.095325    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:26:42.491457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:42.491414    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" event={"ID":"fcf57063-a180-4370-8794-37e58ffbe2e7","Type":"ContainerStarted","Data":"3fe1e1de891ef3ee10c8939ac75e847b5357b9c9a82c0c25c38558831f44e0d3"}
Nov 02 20:26:43.502538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:43.502488    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" event={"ID":"fcf57063-a180-4370-8794-37e58ffbe2e7","Type":"ContainerStarted","Data":"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"}
Nov 02 20:26:43.502991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:43.502590    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:43.614737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:43.614686    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" podStartSLOduration=2.785520927 podStartE2EDuration="3.61467072s" totalImagesPullingTime="829.149793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:43.611916361 +0000 UTC m=+2092.486187248" watchObservedRunningTime="2026-11-02 20:26:43.61467072 +0000 UTC m=+2092.488941605"
Nov 02 20:26:46.095751 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:26:46.095706    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:26:48.047966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:48.047927    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"]
Nov 02 20:26:48.092445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:48.092412    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7432de73-4m5dr"]
Nov 02 20:26:49.339998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:49.339956    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291edc28-9a38-4f22-a0c3-5db00c2e1959" path="/var/lib/kubelet/pods/291edc28-9a38-4f22-a0c3-5db00c2e1959/volumes"
Nov 02 20:26:49.633420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:49.625175    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:26:51.814061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:51.814025    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:26:51.825889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:26:51.825830    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:27:01.822124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.822020    5099 generic.go:356] "Generic (PLEG): container finished" podID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerID="37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a" exitCode=0
Nov 02 20:27:01.822124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.822090    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerDied","Data":"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"}
Nov 02 20:27:01.822773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.822581    5099 scope.go:122] "RemoveContainer" containerID="37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"
Nov 02 20:27:01.824733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.824690    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:01.824851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.824801    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:01.824928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.824850    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:01.961081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.961040    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:27:01.961296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:01.961145    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:27:02.832362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.832271    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerStarted","Data":"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"}
Nov 02 20:27:02.834168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.834136    5099 generic.go:356] "Generic (PLEG): container finished" podID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerID="df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b" exitCode=0
Nov 02 20:27:02.834291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.834202    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerDied","Data":"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"}
Nov 02 20:27:02.834536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.834522    5099 scope.go:122] "RemoveContainer" containerID="df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"
Nov 02 20:27:02.836121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.836092    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:02.836216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.836158    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:02.836216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.836191    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:02.875550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.875520    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:27:02.875765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.875742    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:02.875885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:02.875828    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:03.892813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:03.892026    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:03.892813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:03.892103    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:03.916226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:03.916183    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerStarted","Data":"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"}
Nov 02 20:27:04.777809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:04.775018    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:27:04.777809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:04.775560    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:04.777809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:04.775626    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:04.882071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:04.882033    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:04.882389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:04.882355    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:05.777806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:05.777209    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:05.777806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:05.777272    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:05.878806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:05.878276    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:05.878806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:05.878358    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:06.799135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:06.797288    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:06.799135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:06.797360    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:06.880802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:06.880166    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:06.880802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:06.880242    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:07.778806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:07.777287    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:07.778806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:07.777388    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:07.876274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:07.876234    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 02 20:27:07.876931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:07.876895    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 02 20:27:08.774230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:08.774190    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:08.774539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:08.774507    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:08.968004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:08.967970    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:27:08.970692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:08.970670    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:27:08.984690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:08.984663    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:27:09.774015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:09.773982    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 20:27:09.774229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:09.774040    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 20:27:10.779053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:10.779018    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:27:10.779573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:10.779555    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:27:10.780512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:10.780468    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:27:16.097591 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:27:16.097550    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:27:46.116236 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:27:46.116194    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:27:55.074679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.074579    5099 scope.go:122] "RemoveContainer" containerID="5bc5a5b8101c9711a7d447522fd1aa431cace89158e3bf1228c6936c24a3eff8"
Nov 02 20:27:55.695143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.695110    5099 generic.go:356] "Generic (PLEG): container finished" podID="996ed531-bd0a-4227-9082-79327656bc37" containerID="ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a" exitCode=0
Nov 02 20:27:55.695326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.695180    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerDied","Data":"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"}
Nov 02 20:27:55.695624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.695608    5099 scope.go:122] "RemoveContainer" containerID="ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"
Nov 02 20:27:55.697527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.697499    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:55.697633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.697568    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:55.697633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:55.697599    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:27:56.707019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:56.706981    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerStarted","Data":"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"}
Nov 02 20:27:57.622854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:57.622770    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:27:57.624394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:57.624365    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:27:57.624525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:57.624428    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:27:57.655810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:57.653897    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:27:58.622657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:58.622623    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:27:58.623102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:58.622676    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:27:59.623220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:59.623180    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:27:59.623649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:27:59.623246    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:28:00.622448 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:00.622410    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:28:00.622641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:00.622479    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:28:01.471381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.471339    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"]
Nov 02 20:28:01.476689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.476660    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.505152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.505124    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"]
Nov 02 20:28:01.571127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.571091    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:28:01.571421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.571382    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:28:01.600735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.600700    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.600909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.600740    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.600909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.600793    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2c4\" (UniqueName: \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.623261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.623223    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:28:01.623424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.623293    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:28:01.702121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.702083    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.702317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.702128    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.702317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.702166    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2c4\" (UniqueName: \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.702907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.702882    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.704824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.704799    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.767604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.767521    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2c4\" (UniqueName: \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\") pod \"maas-api-revoke-keys-e2e-api-a-81e58f-tchcl\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.793264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.793234    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:01.976588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.976529    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"]
Nov 02 20:28:01.990259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:01.988275    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.010276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.010244    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"]
Nov 02 20:28:02.110404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.110374    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.110601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.110415    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxm6\" (UniqueName: \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.110601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.110585    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.212186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.212007    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.212186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.212148    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxm6\" (UniqueName: \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.212374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.212280    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.213058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.212680    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.213058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.212962    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"]
Nov 02 20:28:02.214908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.214883    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.271549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.271518    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxm6\" (UniqueName: \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\") pod \"maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.319166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.318736    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:02.623258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.623182    5099 patch_prober.go:29] interesting pod/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 20:28:02.623258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.623232    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 20:28:02.786510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.786472    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl" event={"ID":"0850b6f7-f3ae-41dd-8824-aaacabc84bef","Type":"ContainerStarted","Data":"a29ac17c8f5d62c94555745735b635384b4ceb66d38998b49232c160efe9c7b4"}
Nov 02 20:28:02.842433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:02.842394    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"]
Nov 02 20:28:03.627467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.627441    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:28:03.628635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.628614    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:28:03.796107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.796068    5099 generic.go:356] "Generic (PLEG): container finished" podID="0850b6f7-f3ae-41dd-8824-aaacabc84bef" containerID="d79751dd14d636a30dc7b1bcdd77db89da707102522384270493981dd0c53b26" exitCode=0
Nov 02 20:28:03.796275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.796149    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl" event={"ID":"0850b6f7-f3ae-41dd-8824-aaacabc84bef","Type":"ContainerDied","Data":"d79751dd14d636a30dc7b1bcdd77db89da707102522384270493981dd0c53b26"}
Nov 02 20:28:03.797982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.797962    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv" event={"ID":"60ef1507-c70f-49aa-af51-7bb5bbffa55d","Type":"ContainerStarted","Data":"1565889a5e3e33e18563a4a79512c4a3f78773ba447285423993c3ddee67dd28"}
Nov 02 20:28:03.798068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:03.798001    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv" event={"ID":"60ef1507-c70f-49aa-af51-7bb5bbffa55d","Type":"ContainerStarted","Data":"c9c989bb7bd879defae5ddbe5c5b56b2f5a2d76959938d7a40f0aa49731a1609"}
Nov 02 20:28:04.134833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:04.134755    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv" podStartSLOduration=3.134741276 podStartE2EDuration="3.134741276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:04.025387231 +0000 UTC m=+2172.899658120" watchObservedRunningTime="2026-11-02 20:28:04.134741276 +0000 UTC m=+2173.009012163"
Nov 02 20:28:04.809316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:04.809276    5099 generic.go:356] "Generic (PLEG): container finished" podID="60ef1507-c70f-49aa-af51-7bb5bbffa55d" containerID="1565889a5e3e33e18563a4a79512c4a3f78773ba447285423993c3ddee67dd28" exitCode=0
Nov 02 20:28:04.810095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:04.810064    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv" event={"ID":"60ef1507-c70f-49aa-af51-7bb5bbffa55d","Type":"ContainerDied","Data":"1565889a5e3e33e18563a4a79512c4a3f78773ba447285423993c3ddee67dd28"}
Nov 02 20:28:05.020270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.020241    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:05.082114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.082034    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\") pod \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") "
Nov 02 20:28:05.082288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.082163    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\" (UniqueName: \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\") pod \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") "
Nov 02 20:28:05.082288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.082250    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\") pod \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\" (UID: \"0850b6f7-f3ae-41dd-8824-aaacabc84bef\") "
Nov 02 20:28:05.082668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.082633    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca" pod "0850b6f7-f3ae-41dd-8824-aaacabc84bef" (UID: "0850b6f7-f3ae-41dd-8824-aaacabc84bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:05.083091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.083070    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:05.084836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.084810    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca" pod "0850b6f7-f3ae-41dd-8824-aaacabc84bef" (UID: "0850b6f7-f3ae-41dd-8824-aaacabc84bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:28:05.084941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.084854    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4" pod "0850b6f7-f3ae-41dd-8824-aaacabc84bef" (UID: "0850b6f7-f3ae-41dd-8824-aaacabc84bef"). InnerVolumeSpecName "kube-api-access-dx2c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:05.184666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.184632    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0850b6f7-f3ae-41dd-8824-aaacabc84bef-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:05.184863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.184669    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx2c4\" (UniqueName: \"kubernetes.io/projected/0850b6f7-f3ae-41dd-8824-aaacabc84bef-kube-api-access-dx2c4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:05.819568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.819540    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"
Nov 02 20:28:05.820088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.819582    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl" event={"ID":"0850b6f7-f3ae-41dd-8824-aaacabc84bef","Type":"ContainerDied","Data":"a29ac17c8f5d62c94555745735b635384b4ceb66d38998b49232c160efe9c7b4"}
Nov 02 20:28:05.820088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.819627    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a29ac17c8f5d62c94555745735b635384b4ceb66d38998b49232c160efe9c7b4"
Nov 02 20:28:05.984890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:05.984864    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:06.101618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.101531    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\" (UniqueName: \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\") pod \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") "
Nov 02 20:28:06.101618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.101582    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\") pod \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") "
Nov 02 20:28:06.101909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.101636    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\") pod \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\" (UID: \"60ef1507-c70f-49aa-af51-7bb5bbffa55d\") "
Nov 02 20:28:06.103051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.102182    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca" pod "60ef1507-c70f-49aa-af51-7bb5bbffa55d" (UID: "60ef1507-c70f-49aa-af51-7bb5bbffa55d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:06.104692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.104659    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca" pod "60ef1507-c70f-49aa-af51-7bb5bbffa55d" (UID: "60ef1507-c70f-49aa-af51-7bb5bbffa55d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:28:06.106302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.106273    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:06.106398 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.106307    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ef1507-c70f-49aa-af51-7bb5bbffa55d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:06.107570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.107544    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6" pod "60ef1507-c70f-49aa-af51-7bb5bbffa55d" (UID: "60ef1507-c70f-49aa-af51-7bb5bbffa55d"). InnerVolumeSpecName "kube-api-access-dfxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:06.208206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.208176    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxm6\" (UniqueName: \"kubernetes.io/projected/60ef1507-c70f-49aa-af51-7bb5bbffa55d-kube-api-access-dfxm6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:06.829506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.829477    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"
Nov 02 20:28:06.829946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.829514    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv" event={"ID":"60ef1507-c70f-49aa-af51-7bb5bbffa55d","Type":"ContainerDied","Data":"c9c989bb7bd879defae5ddbe5c5b56b2f5a2d76959938d7a40f0aa49731a1609"}
Nov 02 20:28:06.829946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:06.829559    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c989bb7bd879defae5ddbe5c5b56b2f5a2d76959938d7a40f0aa49731a1609"
Nov 02 20:28:09.912921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:09.912884    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:28:12.432760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:12.432710    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:28:12.543927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:12.543889    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:28:14.913826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:14.913755    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerName="payload-pre-processing" containerID="cri-o://7e3f79fe3ca4e99392eade93df3c07678e160874f91cbca5a93b60f4f5964e8c" gracePeriod=25
Nov 02 20:28:15.042822 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:15.042795    5099 logging.go:55] [core] [Channel #6740 SubChannel #6741]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.202:50052", ServerName: "10.132.0.202:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.202:50052: connect: connection refused"
Nov 02 20:28:15.071178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.071147    5099 generic.go:356] "Generic (PLEG): container finished" podID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerID="7e3f79fe3ca4e99392eade93df3c07678e160874f91cbca5a93b60f4f5964e8c" exitCode=0
Nov 02 20:28:15.071356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.071229    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" event={"ID":"4aab373a-5fe7-4192-b54c-2de3093b0284","Type":"ContainerDied","Data":"7e3f79fe3ca4e99392eade93df3c07678e160874f91cbca5a93b60f4f5964e8c"}
Nov 02 20:28:15.227249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.227219    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:28:15.287425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.287395    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\") pod \"4aab373a-5fe7-4192-b54c-2de3093b0284\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") "
Nov 02 20:28:15.287611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.287595    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\" (UniqueName: \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\") pod \"4aab373a-5fe7-4192-b54c-2de3093b0284\" (UID: \"4aab373a-5fe7-4192-b54c-2de3093b0284\") "
Nov 02 20:28:15.287912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.287883    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume" pod "4aab373a-5fe7-4192-b54c-2de3093b0284" (UID: "4aab373a-5fe7-4192-b54c-2de3093b0284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:15.288517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.288489    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aab373a-5fe7-4192-b54c-2de3093b0284-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:15.290161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.290132    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77" pod "4aab373a-5fe7-4192-b54c-2de3093b0284" (UID: "4aab373a-5fe7-4192-b54c-2de3093b0284"). InnerVolumeSpecName "kube-api-access-5gp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:15.391825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.391792    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp77\" (UniqueName: \"kubernetes.io/projected/4aab373a-5fe7-4192-b54c-2de3093b0284-kube-api-access-5gp77\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:15.392072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.392042    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"]
Nov 02 20:28:15.393467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393438    5099 cpu_manager.go:414] "containerMap: removing container" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerName="payload-pre-processing"
Nov 02 20:28:15.393467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393466    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerName="payload-pre-processing"
Nov 02 20:28:15.393646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393500    5099 cpu_manager.go:414] "containerMap: removing container" podUID="60ef1507-c70f-49aa-af51-7bb5bbffa55d" containerName="revoke-keys"
Nov 02 20:28:15.393646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393511    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ef1507-c70f-49aa-af51-7bb5bbffa55d" containerName="revoke-keys"
Nov 02 20:28:15.393646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393609    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0850b6f7-f3ae-41dd-8824-aaacabc84bef" containerName="revoke-keys"
Nov 02 20:28:15.393646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393618    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0850b6f7-f3ae-41dd-8824-aaacabc84bef" containerName="revoke-keys"
Nov 02 20:28:15.393858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393845    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerName="payload-pre-processing"
Nov 02 20:28:15.393916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393869    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ef1507-c70f-49aa-af51-7bb5bbffa55d" containerName="revoke-keys"
Nov 02 20:28:15.393916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.393895    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0850b6f7-f3ae-41dd-8824-aaacabc84bef" containerName="revoke-keys"
Nov 02 20:28:15.399110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.398585    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.420605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.419077    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"]
Nov 02 20:28:15.448355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.448268    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:28:15.448355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.448321    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:28:15.494673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.494521    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.494673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.494648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.494959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.494738    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nhb\" (UniqueName: \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.596269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.596235    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.596451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.596301    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.596451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.596363    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nhb\" (UniqueName: \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.597132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.597108    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.598676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.598650    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.653907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.653870    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nhb\" (UniqueName: \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\") pod \"maas-api-revoke-keys-e2e-api-a-c76017-h9wsn\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:15.722101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:15.722018    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:16.043382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.043288    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.202:50052\" within 1s: context deadline exceeded"
Nov 02 20:28:16.043819 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:16.043387    5099 logging.go:55] [core] [Channel #6740 SubChannel #6741]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.202:50052", ServerName: "10.132.0.202:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.202:50052: operation was canceled"
Nov 02 20:28:16.086801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.086749    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"
Nov 02 20:28:16.087005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.086771    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb" event={"ID":"4aab373a-5fe7-4192-b54c-2de3093b0284","Type":"ContainerDied","Data":"2a29366edfc3a2ecf4f67828f39fb14dbd610c9e415abb6917204d42f4da6d44"}
Nov 02 20:28:16.087005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.086890    5099 scope.go:122] "RemoveContainer" containerID="7e3f79fe3ca4e99392eade93df3c07678e160874f91cbca5a93b60f4f5964e8c"
Nov 02 20:28:16.098349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.098308    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"]
Nov 02 20:28:16.100852 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:16.100823    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8257ab67_825f_4457_b8f3_06d5069ea61d.slice/crio-f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b WatchSource:0}: Error finding container f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b: Status 404 returned error can't find the container with id f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b
Nov 02 20:28:16.118330 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:16.118300    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:28:16.245940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.245908    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:28:16.250968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:16.250941    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81e58f-7cfd687b75-xn8hb"]
Nov 02 20:28:17.098197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.098154    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn" event={"ID":"8257ab67-825f-4457-b8f3-06d5069ea61d","Type":"ContainerStarted","Data":"651e2aede76dd08feb520706fb3155fb9c1af259ff24b8eb2a402f914c178915"}
Nov 02 20:28:17.098615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.098263    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn" event={"ID":"8257ab67-825f-4457-b8f3-06d5069ea61d","Type":"ContainerStarted","Data":"f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b"}
Nov 02 20:28:17.211771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.211544    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn" podStartSLOduration=2.211523933 podStartE2EDuration="2.211523933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:17.207236737 +0000 UTC m=+2186.081507626" watchObservedRunningTime="2026-11-02 20:28:17.211523933 +0000 UTC m=+2186.085794820"
Nov 02 20:28:17.242214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.242185    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"]
Nov 02 20:28:17.247084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.247063    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.261480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.261454    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aab373a-5fe7-4192-b54c-2de3093b0284" path="/var/lib/kubelet/pods/4aab373a-5fe7-4192-b54c-2de3093b0284/volumes"
Nov 02 20:28:17.314332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.314280    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"]
Nov 02 20:28:17.342598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.342566    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.342775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.342617    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5p4\" (UniqueName: \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.343244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.342953    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.434141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.434100    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" podUID="fad19e3a-d94a-481c-8608-31ae0115387a" containerName="payload-processing" containerID="cri-o://130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2" gracePeriod=25
Nov 02 20:28:17.445015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.444983    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.445149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.445112    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.445283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.445165    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5p4\" (UniqueName: \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.445983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.445953    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.447468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.447440    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.499042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.499015    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5p4\" (UniqueName: \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\") pod \"maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.544751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.544705    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" podUID="affab4fc-381b-4f15-862b-393e830cb029" containerName="payload-pre-processing" containerID="cri-o://545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4" gracePeriod=25
Nov 02 20:28:17.603101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.603062    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:17.736167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.735362    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:28:17.852395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.852362    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\" (UniqueName: \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\") pod \"fad19e3a-d94a-481c-8608-31ae0115387a\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") "
Nov 02 20:28:17.852587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.852414    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\") pod \"fad19e3a-d94a-481c-8608-31ae0115387a\" (UID: \"fad19e3a-d94a-481c-8608-31ae0115387a\") "
Nov 02 20:28:17.855960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.855923    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz" pod "fad19e3a-d94a-481c-8608-31ae0115387a" (UID: "fad19e3a-d94a-481c-8608-31ae0115387a"). InnerVolumeSpecName "kube-api-access-j72tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:17.858094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.858063    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume" pod "fad19e3a-d94a-481c-8608-31ae0115387a" (UID: "fad19e3a-d94a-481c-8608-31ae0115387a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:17.895156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.895128    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:28:17.955201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.955123    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j72tz\" (UniqueName: \"kubernetes.io/projected/fad19e3a-d94a-481c-8608-31ae0115387a-kube-api-access-j72tz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:17.955201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:17.955149    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad19e3a-d94a-481c-8608-31ae0115387a-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:18.016523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.016475    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"]
Nov 02 20:28:18.053095 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:18.053059    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82e979a_2479_420a_abfe_8e7f80b746c1.slice/crio-16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6 WatchSource:0}: Error finding container 16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6: Status 404 returned error can't find the container with id 16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6
Nov 02 20:28:18.056307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.056275    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\" (UniqueName: \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\") pod \"affab4fc-381b-4f15-862b-393e830cb029\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") "
Nov 02 20:28:18.056435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.056353    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\") pod \"affab4fc-381b-4f15-862b-393e830cb029\" (UID: \"affab4fc-381b-4f15-862b-393e830cb029\") "
Nov 02 20:28:18.057316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.057279    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume" pod "affab4fc-381b-4f15-862b-393e830cb029" (UID: "affab4fc-381b-4f15-862b-393e830cb029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:18.059432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.059407    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6" pod "affab4fc-381b-4f15-862b-393e830cb029" (UID: "affab4fc-381b-4f15-862b-393e830cb029"). InnerVolumeSpecName "kube-api-access-btnj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:18.108953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.108925    5099 generic.go:356] "Generic (PLEG): container finished" podID="8257ab67-825f-4457-b8f3-06d5069ea61d" containerID="651e2aede76dd08feb520706fb3155fb9c1af259ff24b8eb2a402f914c178915" exitCode=0
Nov 02 20:28:18.109296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.108985    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn" event={"ID":"8257ab67-825f-4457-b8f3-06d5069ea61d","Type":"ContainerDied","Data":"651e2aede76dd08feb520706fb3155fb9c1af259ff24b8eb2a402f914c178915"}
Nov 02 20:28:18.110613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.110590    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2" event={"ID":"b82e979a-2479-420a-abfe-8e7f80b746c1","Type":"ContainerStarted","Data":"16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6"}
Nov 02 20:28:18.112117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.112094    5099 generic.go:356] "Generic (PLEG): container finished" podID="fad19e3a-d94a-481c-8608-31ae0115387a" containerID="130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2" exitCode=0
Nov 02 20:28:18.112225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.112136    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"
Nov 02 20:28:18.112225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.112126    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" event={"ID":"fad19e3a-d94a-481c-8608-31ae0115387a","Type":"ContainerDied","Data":"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"}
Nov 02 20:28:18.112336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.112263    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t" event={"ID":"fad19e3a-d94a-481c-8608-31ae0115387a","Type":"ContainerDied","Data":"12d8c60c7c93218cb545d03977f8dffe01c2cd734a42ffef5a490ea380fdc1ee"}
Nov 02 20:28:18.112336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.112281    5099 scope.go:122] "RemoveContainer" containerID="130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"
Nov 02 20:28:18.113711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.113685    5099 generic.go:356] "Generic (PLEG): container finished" podID="affab4fc-381b-4f15-862b-393e830cb029" containerID="545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4" exitCode=0
Nov 02 20:28:18.113830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.113720    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" event={"ID":"affab4fc-381b-4f15-862b-393e830cb029","Type":"ContainerDied","Data":"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"}
Nov 02 20:28:18.113830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.113734    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"
Nov 02 20:28:18.113830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.113756    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc" event={"ID":"affab4fc-381b-4f15-862b-393e830cb029","Type":"ContainerDied","Data":"7a6ec76b3676d0b03fc1a1cb0250b27e9e03685d0e09ef15b9d8da4ff625cd01"}
Nov 02 20:28:18.126027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.126005    5099 scope.go:122] "RemoveContainer" containerID="130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"
Nov 02 20:28:18.126316 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:18.126282    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2\": container with ID starting with 130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2 not found: ID does not exist" containerID="130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"
Nov 02 20:28:18.126404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.126383    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2"} err="failed to get container status \"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2\": rpc error: code = NotFound desc = could not find container \"130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2\": container with ID starting with 130549725a4668d26414ada9360ee61cab86c5180e17d2297c8b8a04b042a7c2 not found: ID does not exist"
Nov 02 20:28:18.126463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.126407    5099 scope.go:122] "RemoveContainer" containerID="545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"
Nov 02 20:28:18.136810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.136773    5099 scope.go:122] "RemoveContainer" containerID="545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"
Nov 02 20:28:18.137102 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:18.137076    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4\": container with ID starting with 545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4 not found: ID does not exist" containerID="545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"
Nov 02 20:28:18.137202 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.137106    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4"} err="failed to get container status \"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4\": rpc error: code = NotFound desc = could not find container \"545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4\": container with ID starting with 545d651ef1fc3e5b9cdaff05617ceab1e2502a594f888df7e6a7c4a3adadbeb4 not found: ID does not exist"
Nov 02 20:28:18.158386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.158349    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnj6\" (UniqueName: \"kubernetes.io/projected/affab4fc-381b-4f15-862b-393e830cb029-kube-api-access-btnj6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:18.158386 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.158381    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affab4fc-381b-4f15-862b-393e830cb029-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:18.482858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.479207    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:28:18.548211 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.548183    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7dcb03-7f94c85655-7sf4t"]
Nov 02 20:28:18.559404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.559375    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:28:18.566964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:18.566923    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7dcb03-98f4495cf-ss2pc"]
Nov 02 20:28:19.173952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.165801    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2" event={"ID":"b82e979a-2479-420a-abfe-8e7f80b746c1","Type":"ContainerStarted","Data":"d002fbf3ec32ef976590a8d1a7d103cb971b1a05950e1803c6c5b6ab3325b742"}
Nov 02 20:28:19.260606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.260573    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affab4fc-381b-4f15-862b-393e830cb029" path="/var/lib/kubelet/pods/affab4fc-381b-4f15-862b-393e830cb029/volumes"
Nov 02 20:28:19.261273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.261250    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad19e3a-d94a-481c-8608-31ae0115387a" path="/var/lib/kubelet/pods/fad19e3a-d94a-481c-8608-31ae0115387a/volumes"
Nov 02 20:28:19.280440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.280381    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2" podStartSLOduration=2.280364909 podStartE2EDuration="2.280364909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:19.274035583 +0000 UTC m=+2188.148306471" watchObservedRunningTime="2026-11-02 20:28:19.280364909 +0000 UTC m=+2188.154635802"
Nov 02 20:28:19.386324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.386293    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:19.447505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.447406    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\") pod \"8257ab67-825f-4457-b8f3-06d5069ea61d\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") "
Nov 02 20:28:19.447505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.447467    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\") pod \"8257ab67-825f-4457-b8f3-06d5069ea61d\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") "
Nov 02 20:28:19.447740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.447576    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\" (UniqueName: \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\") pod \"8257ab67-825f-4457-b8f3-06d5069ea61d\" (UID: \"8257ab67-825f-4457-b8f3-06d5069ea61d\") "
Nov 02 20:28:19.447912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.447887    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca" pod "8257ab67-825f-4457-b8f3-06d5069ea61d" (UID: "8257ab67-825f-4457-b8f3-06d5069ea61d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:19.448275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.448243    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:19.449837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.449803    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca" pod "8257ab67-825f-4457-b8f3-06d5069ea61d" (UID: "8257ab67-825f-4457-b8f3-06d5069ea61d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:28:19.450069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.450043    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb" pod "8257ab67-825f-4457-b8f3-06d5069ea61d" (UID: "8257ab67-825f-4457-b8f3-06d5069ea61d"). InnerVolumeSpecName "kube-api-access-s9nhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:19.550601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.550559    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8257ab67-825f-4457-b8f3-06d5069ea61d-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:19.550601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:19.550597    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nhb\" (UniqueName: \"kubernetes.io/projected/8257ab67-825f-4457-b8f3-06d5069ea61d-kube-api-access-s9nhb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:20.210791 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:20.210758    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"
Nov 02 20:28:20.211250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:20.210800    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn" event={"ID":"8257ab67-825f-4457-b8f3-06d5069ea61d","Type":"ContainerDied","Data":"f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b"}
Nov 02 20:28:20.211250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:20.210859    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15ed81e94a9e44186df4b407090be70139bccc4dbb8f8bf166c69bedf148d7b"
Nov 02 20:28:20.218412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:20.218381    5099 generic.go:356] "Generic (PLEG): container finished" podID="b82e979a-2479-420a-abfe-8e7f80b746c1" containerID="d002fbf3ec32ef976590a8d1a7d103cb971b1a05950e1803c6c5b6ab3325b742" exitCode=0
Nov 02 20:28:20.218585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:20.218497    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2" event={"ID":"b82e979a-2479-420a-abfe-8e7f80b746c1","Type":"ContainerDied","Data":"d002fbf3ec32ef976590a8d1a7d103cb971b1a05950e1803c6c5b6ab3325b742"}
Nov 02 20:28:21.407288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.407260    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:21.480832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.480724    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\") pod \"b82e979a-2479-420a-abfe-8e7f80b746c1\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") "
Nov 02 20:28:21.481019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.480971    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\") pod \"b82e979a-2479-420a-abfe-8e7f80b746c1\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") "
Nov 02 20:28:21.481086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.481052    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\" (UniqueName: \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\") pod \"b82e979a-2479-420a-abfe-8e7f80b746c1\" (UID: \"b82e979a-2479-420a-abfe-8e7f80b746c1\") "
Nov 02 20:28:21.481460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.481400    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca" pod "b82e979a-2479-420a-abfe-8e7f80b746c1" (UID: "b82e979a-2479-420a-abfe-8e7f80b746c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:21.481944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.481919    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:21.483120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.483091    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca" pod "b82e979a-2479-420a-abfe-8e7f80b746c1" (UID: "b82e979a-2479-420a-abfe-8e7f80b746c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:28:21.483292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.483269    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4" pod "b82e979a-2479-420a-abfe-8e7f80b746c1" (UID: "b82e979a-2479-420a-abfe-8e7f80b746c1"). InnerVolumeSpecName "kube-api-access-8s5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:21.583629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.583590    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e979a-2479-420a-abfe-8e7f80b746c1-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:21.583629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:21.583628    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5p4\" (UniqueName: \"kubernetes.io/projected/b82e979a-2479-420a-abfe-8e7f80b746c1-kube-api-access-8s5p4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:22.238031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.237995    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2" event={"ID":"b82e979a-2479-420a-abfe-8e7f80b746c1","Type":"ContainerDied","Data":"16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6"}
Nov 02 20:28:22.238031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.238019    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"
Nov 02 20:28:22.238285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.238048    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16d7ec6f4110ec7c5df52104b0165418e47f6fc73b0d0e04ee385498d09035e6"
Nov 02 20:28:22.753268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.753225    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:28:22.754724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754696    5099 cpu_manager.go:414] "containerMap: removing container" podUID="affab4fc-381b-4f15-862b-393e830cb029" containerName="payload-pre-processing"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754725    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affab4fc-381b-4f15-862b-393e830cb029" containerName="payload-pre-processing"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754760    5099 cpu_manager.go:414] "containerMap: removing container" podUID="fad19e3a-d94a-481c-8608-31ae0115387a" containerName="payload-processing"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754769    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad19e3a-d94a-481c-8608-31ae0115387a" containerName="payload-processing"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754796    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b82e979a-2479-420a-abfe-8e7f80b746c1" containerName="revoke-keys"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754803    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82e979a-2479-420a-abfe-8e7f80b746c1" containerName="revoke-keys"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754844    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8257ab67-825f-4457-b8f3-06d5069ea61d" containerName="revoke-keys"
Nov 02 20:28:22.754881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.754852    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8257ab67-825f-4457-b8f3-06d5069ea61d" containerName="revoke-keys"
Nov 02 20:28:22.755107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.755058    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8257ab67-825f-4457-b8f3-06d5069ea61d" containerName="revoke-keys"
Nov 02 20:28:22.755107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.755081    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="affab4fc-381b-4f15-862b-393e830cb029" containerName="payload-pre-processing"
Nov 02 20:28:22.755107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.755097    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad19e3a-d94a-481c-8608-31ae0115387a" containerName="payload-processing"
Nov 02 20:28:22.755107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.755107    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82e979a-2479-420a-abfe-8e7f80b746c1" containerName="revoke-keys"
Nov 02 20:28:22.758805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.758768    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803173    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803219    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803348    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803474    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803520    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803624    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803692    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7lf\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803798    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.803906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.803893    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.854940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.854902    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:28:22.905024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.904991    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905135    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905165    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905229    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905310    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905366    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905417    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905482    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7lf\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905560    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.905818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.905773    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.906047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.906015    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.906155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.906084    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.906155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.906083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.906437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.906412    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.908097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.908076    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.908344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.908321    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.960271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.960236    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:22.973799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:22.973755    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7lf\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:23.074904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:23.074801    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:23.499442 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:23.499413    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9a2dff_eecf_41b6_aeb8_f5d621941e63.slice/crio-c22fe53ef60c3e4604094ba7e6b56d4e604ad619b97d32a64b81a0dcc0679940 WatchSource:0}: Error finding container c22fe53ef60c3e4604094ba7e6b56d4e604ad619b97d32a64b81a0dcc0679940: Status 404 returned error can't find the container with id c22fe53ef60c3e4604094ba7e6b56d4e604ad619b97d32a64b81a0dcc0679940
Nov 02 20:28:23.499733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:23.499712    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:28:23.502396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:23.502360    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:23.502504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:23.502442    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:23.502504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:23.502487    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:24.263688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:24.263650    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" event={"ID":"ce9a2dff-eecf-41b6-aeb8-f5d621941e63","Type":"ContainerStarted","Data":"63329d22922f2e3810c3210436236a44b8ef3aca5cc2137eb87ec9fb2577638e"}
Nov 02 20:28:24.264217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:24.263709    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" event={"ID":"ce9a2dff-eecf-41b6-aeb8-f5d621941e63","Type":"ContainerStarted","Data":"c22fe53ef60c3e4604094ba7e6b56d4e604ad619b97d32a64b81a0dcc0679940"}
Nov 02 20:28:24.378426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:24.378367    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podStartSLOduration=2.378353027 podStartE2EDuration="2.378353027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:24.373757556 +0000 UTC m=+2193.248028446" watchObservedRunningTime="2026-11-02 20:28:24.378353027 +0000 UTC m=+2193.252623915"
Nov 02 20:28:25.075456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:25.075419    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:25.077317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:25.077289    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:25.077465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:25.077362    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:26.076025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:26.075983    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:26.076558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:26.076060    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:26.905654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:26.905618    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:28:27.027929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:27.027896    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:28:27.075458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:27.075416    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:27.075640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:27.075487    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:28.075221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:28.075181    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:28.075749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:28.075236    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:29.077191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.077152    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:29.077773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.077741    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:29.433421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.433375    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"]
Nov 02 20:28:29.451121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.451087    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.489945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.489909    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"]
Nov 02 20:28:29.528491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.528451    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.528766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.528517    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.528766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.528673    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfts\" (UniqueName: \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.531935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.531901    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:28:29.532057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.532000    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:28:29.630075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.630034    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.630305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.630099    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.630305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.630169    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfts\" (UniqueName: \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.630906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.630881    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.632573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.632548    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.686907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.686777    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfts\" (UniqueName: \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\") pod \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:29.782228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:29.782189    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:30.075825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:30.075718    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 20:28:30.075825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:30.075808    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 20:28:30.211518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:30.211499    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"]
Nov 02 20:28:30.211939 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:30.211905    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea256302_5170_40d9_a7f1_996a87540a6b.slice/crio-8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159 WatchSource:0}: Error finding container 8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159: Status 404 returned error can't find the container with id 8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159
Nov 02 20:28:30.365996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:30.365941    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt" event={"ID":"ea256302-5170-40d9-a7f1-996a87540a6b","Type":"ContainerStarted","Data":"8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159"}
Nov 02 20:28:31.081643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.081567    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:31.082046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.082022    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:31.082749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.082728    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:28:31.406541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.406507    5099 generic.go:356] "Generic (PLEG): container finished" podID="ea256302-5170-40d9-a7f1-996a87540a6b" containerID="15ad3f0f9a839e2dc1af8d0d3b1437b50808cadbf3f99737252e6bd44980e522" exitCode=0
Nov 02 20:28:31.407038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.406610    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt" event={"ID":"ea256302-5170-40d9-a7f1-996a87540a6b","Type":"ContainerDied","Data":"15ad3f0f9a839e2dc1af8d0d3b1437b50808cadbf3f99737252e6bd44980e522"}
Nov 02 20:28:31.906872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:31.906827    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" podUID="9ccd97a7-719b-4832-b85e-e7c0728f276a" containerName="payload-processing" containerID="cri-o://524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c" gracePeriod=25
Nov 02 20:28:32.028572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.028529    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" podUID="07d390e7-494a-4580-978d-c3fc3e62b572" containerName="payload-pre-processing" containerID="cri-o://41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b" gracePeriod=25
Nov 02 20:28:32.226898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.226856    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:28:32.389553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.389161    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:28:32.417320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.417219    5099 generic.go:356] "Generic (PLEG): container finished" podID="07d390e7-494a-4580-978d-c3fc3e62b572" containerID="41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b" exitCode=0
Nov 02 20:28:32.417320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.417264    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"
Nov 02 20:28:32.417320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.417312    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" event={"ID":"07d390e7-494a-4580-978d-c3fc3e62b572","Type":"ContainerDied","Data":"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"}
Nov 02 20:28:32.417974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.417367    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4" event={"ID":"07d390e7-494a-4580-978d-c3fc3e62b572","Type":"ContainerDied","Data":"5d2e6d7bf10e6cac098738f298ea81e409cf35d2010a3da813f27b781e5c9d8a"}
Nov 02 20:28:32.417974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.417396    5099 scope.go:122] "RemoveContainer" containerID="41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"
Nov 02 20:28:32.418587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.418553    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\" (UniqueName: \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\") pod \"07d390e7-494a-4580-978d-c3fc3e62b572\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") "
Nov 02 20:28:32.418677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.418661    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\") pod \"07d390e7-494a-4580-978d-c3fc3e62b572\" (UID: \"07d390e7-494a-4580-978d-c3fc3e62b572\") "
Nov 02 20:28:32.418767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.418749    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\" (UniqueName: \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\") pod \"9ccd97a7-719b-4832-b85e-e7c0728f276a\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") "
Nov 02 20:28:32.418842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.418796    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\") pod \"9ccd97a7-719b-4832-b85e-e7c0728f276a\" (UID: \"9ccd97a7-719b-4832-b85e-e7c0728f276a\") "
Nov 02 20:28:32.419228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419202    5099 generic.go:356] "Generic (PLEG): container finished" podID="9ccd97a7-719b-4832-b85e-e7c0728f276a" containerID="524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c" exitCode=0
Nov 02 20:28:32.419330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419304    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume" pod "07d390e7-494a-4580-978d-c3fc3e62b572" (UID: "07d390e7-494a-4580-978d-c3fc3e62b572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:32.419400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419344    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"
Nov 02 20:28:32.419400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419374    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" event={"ID":"9ccd97a7-719b-4832-b85e-e7c0728f276a","Type":"ContainerDied","Data":"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"}
Nov 02 20:28:32.419521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419434    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl" event={"ID":"9ccd97a7-719b-4832-b85e-e7c0728f276a","Type":"ContainerDied","Data":"93ba7075fd80b493c50a9865097bd6338081d13ef06a99697d5c5fd35b515f59"}
Nov 02 20:28:32.419659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419638    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume" pod "9ccd97a7-719b-4832-b85e-e7c0728f276a" (UID: "9ccd97a7-719b-4832-b85e-e7c0728f276a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:32.419727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.419656    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d390e7-494a-4580-978d-c3fc3e62b572-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:32.421821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.421796    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh" pod "07d390e7-494a-4580-978d-c3fc3e62b572" (UID: "07d390e7-494a-4580-978d-c3fc3e62b572"). InnerVolumeSpecName "kube-api-access-c52bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:32.422020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.422000    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h" pod "9ccd97a7-719b-4832-b85e-e7c0728f276a" (UID: "9ccd97a7-719b-4832-b85e-e7c0728f276a"). InnerVolumeSpecName "kube-api-access-6zx8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:32.429814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.429773    5099 scope.go:122] "RemoveContainer" containerID="41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"
Nov 02 20:28:32.430170 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:32.430149    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b\": container with ID starting with 41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b not found: ID does not exist" containerID="41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"
Nov 02 20:28:32.430245 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.430184    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b"} err="failed to get container status \"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b\": rpc error: code = NotFound desc = could not find container \"41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b\": container with ID starting with 41a2a0c407e113eeb432b5bb59f9f2e85dcf5867a2f216bbe775b17a2f2e5b8b not found: ID does not exist"
Nov 02 20:28:32.430245 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.430208    5099 scope.go:122] "RemoveContainer" containerID="524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"
Nov 02 20:28:32.448658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.448627    5099 scope.go:122] "RemoveContainer" containerID="524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"
Nov 02 20:28:32.454970 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:32.454759    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c\": container with ID starting with 524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c not found: ID does not exist" containerID="524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"
Nov 02 20:28:32.454970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.454841    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c"} err="failed to get container status \"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c\": rpc error: code = NotFound desc = could not find container \"524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c\": container with ID starting with 524ac204cbdae4af8deddc233ff5ab4074b7db42ea93bb59a52f4227cc83136c not found: ID does not exist"
Nov 02 20:28:32.601999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.586441    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52bh\" (UniqueName: \"kubernetes.io/projected/07d390e7-494a-4580-978d-c3fc3e62b572-kube-api-access-c52bh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:32.601999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.586475    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zx8h\" (UniqueName: \"kubernetes.io/projected/9ccd97a7-719b-4832-b85e-e7c0728f276a-kube-api-access-6zx8h\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:32.601999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:32.586493    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ccd97a7-719b-4832-b85e-e7c0728f276a-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:33.910668 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:33.909113    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 20:28:33.916191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:33.915909    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 20:28:33.938671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:33.938639    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:28:33.938860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:33.938690    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c76017-f5894596c-p2qsl"]
Nov 02 20:28:33.940544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:33.940522    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:34.009773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.007277    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:28:34.018945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.018905    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\" (UniqueName: \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\") pod \"ea256302-5170-40d9-a7f1-996a87540a6b\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") "
Nov 02 20:28:34.019137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.018966    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\") pod \"ea256302-5170-40d9-a7f1-996a87540a6b\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") "
Nov 02 20:28:34.019226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.019188    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\") pod \"ea256302-5170-40d9-a7f1-996a87540a6b\" (UID: \"ea256302-5170-40d9-a7f1-996a87540a6b\") "
Nov 02 20:28:34.021378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.021342    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca" pod "ea256302-5170-40d9-a7f1-996a87540a6b" (UID: "ea256302-5170-40d9-a7f1-996a87540a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:28:34.024917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.024479    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts" pod "ea256302-5170-40d9-a7f1-996a87540a6b" (UID: "ea256302-5170-40d9-a7f1-996a87540a6b"). InnerVolumeSpecName "kube-api-access-5xfts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:28:34.025836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.025758    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca" pod "ea256302-5170-40d9-a7f1-996a87540a6b" (UID: "ea256302-5170-40d9-a7f1-996a87540a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:28:34.031952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.031920    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c76017-86d6b7d59-tsnh4"]
Nov 02 20:28:34.122204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.122150    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:34.122204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.122199    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfts\" (UniqueName: \"kubernetes.io/projected/ea256302-5170-40d9-a7f1-996a87540a6b-kube-api-access-5xfts\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:34.122204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.122211    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea256302-5170-40d9-a7f1-996a87540a6b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:28:34.949154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.949123    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:34.949681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.949152    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt" event={"ID":"ea256302-5170-40d9-a7f1-996a87540a6b","Type":"ContainerDied","Data":"8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159"}
Nov 02 20:28:34.949681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:34.949198    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d4e895e88b3c9a37a6c2204f0503d46f7684e959afccd938e4e6ea6d4372159"
Nov 02 20:28:35.261469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:35.261392    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d390e7-494a-4580-978d-c3fc3e62b572" path="/var/lib/kubelet/pods/07d390e7-494a-4580-978d-c3fc3e62b572/volumes"
Nov 02 20:28:35.261969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:35.261952    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccd97a7-719b-4832-b85e-e7c0728f276a" path="/var/lib/kubelet/pods/9ccd97a7-719b-4832-b85e-e7c0728f276a/volumes"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.890154    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891638    5099 cpu_manager.go:414] "containerMap: removing container" podUID="07d390e7-494a-4580-978d-c3fc3e62b572" containerName="payload-pre-processing"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891662    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d390e7-494a-4580-978d-c3fc3e62b572" containerName="payload-pre-processing"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891716    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ea256302-5170-40d9-a7f1-996a87540a6b" containerName="revoke-keys"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891726    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea256302-5170-40d9-a7f1-996a87540a6b" containerName="revoke-keys"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891762    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9ccd97a7-719b-4832-b85e-e7c0728f276a" containerName="payload-processing"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.891769    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccd97a7-719b-4832-b85e-e7c0728f276a" containerName="payload-processing"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.892004    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d390e7-494a-4580-978d-c3fc3e62b572" containerName="payload-pre-processing"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.892025    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea256302-5170-40d9-a7f1-996a87540a6b" containerName="revoke-keys"
Nov 02 20:28:37.895894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.892039    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccd97a7-719b-4832-b85e-e7c0728f276a" containerName="payload-processing"
Nov 02 20:28:37.907843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.903065    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.950885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.946799    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:28:37.962465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.962434    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.962755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.962663    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.962954    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lll7\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.963131    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.963258    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.963383    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.963487    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.963743 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.963725    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:37.964019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:37.964001    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.065567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065525    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.065773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065698    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.065877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065837    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.065945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065884    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.065945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065928    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lll7\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.065965    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066011    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066060    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066093    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066158    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066237    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066509    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066589    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.066933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.066908    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.068625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.068599    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.069173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.069150    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.119585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.119549    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.119717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.119684    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lll7\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.256472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.256378    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:38.683616 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:28:38.683582    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99642394_2af1_4b51_963e_1415018eb80a.slice/crio-66224beed883c35c868e589bb841bcd8165fe700b81a070760e6553e84acb278 WatchSource:0}: Error finding container 66224beed883c35c868e589bb841bcd8165fe700b81a070760e6553e84acb278: Status 404 returned error can't find the container with id 66224beed883c35c868e589bb841bcd8165fe700b81a070760e6553e84acb278
Nov 02 20:28:38.685047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.685014    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:28:38.686550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.686517    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:38.686640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.686595    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:38.686717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:38.686640    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:39.007713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.007620    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" event={"ID":"99642394-2af1-4b51-963e-1415018eb80a","Type":"ContainerStarted","Data":"0151cd3a500871eb14b39f47c53a03b00ffd3fb2bf9c85fd36fab38d968d9b90"}
Nov 02 20:28:39.007713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.007695    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" event={"ID":"99642394-2af1-4b51-963e-1415018eb80a","Type":"ContainerStarted","Data":"66224beed883c35c868e589bb841bcd8165fe700b81a070760e6553e84acb278"}
Nov 02 20:28:39.120433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.120382    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podStartSLOduration=2.120367583 podStartE2EDuration="2.120367583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:39.117975843 +0000 UTC m=+2207.992246732" watchObservedRunningTime="2026-11-02 20:28:39.120367583 +0000 UTC m=+2207.994638466"
Nov 02 20:28:39.259723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.259633    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:39.259723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.259686    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:39.263515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:39.263488    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:40.275485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:40.275445    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:40.276040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:40.275527    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:41.281116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:41.281071    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:41.281815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:41.281749    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:42.282291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:42.282249    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:42.282918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:42.282884    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:43.287265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:43.286823    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:43.287265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:43.286892    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:44.269807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:44.268886    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:44.269807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:44.268952    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:45.279807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:45.278934    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 20:28:45.279807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:45.278998    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 20:28:46.112421 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:46.112380    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:28:46.328518 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:46.328485    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:46.329255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:46.329224    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:46.330242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:46.330217    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:28:49.342956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:49.342913    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:28:49.343399 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:49.342977    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:28:51.220764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.220731    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"]
Nov 02 20:28:51.267384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.267355    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"]
Nov 02 20:28:51.304928 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:28:51.304891    5099 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt\" is forbidden: User \"system:node:ip-10-0-133-26.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-26.us-west-2.compute.internal' and this object" podUID="ea256302-5170-40d9-a7f1-996a87540a6b" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5a51f-fgkkt"
Nov 02 20:28:51.320153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.320124    5099 generic.go:356] "Generic (PLEG): container finished" podID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerID="603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6" exitCode=0
Nov 02 20:28:51.320316 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.320192    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerDied","Data":"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"}
Nov 02 20:28:51.320614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.320595    5099 scope.go:122] "RemoveContainer" containerID="603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"
Nov 02 20:28:51.322327 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.322297    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:51.322455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.322381    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:51.322455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:51.322429    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:28:52.333493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:52.333460    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerStarted","Data":"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"}
Nov 02 20:28:53.135581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:53.135548    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:28:53.135766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:53.135740    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:28:53.135867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:53.135825    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:28:53.284630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:53.284585    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea256302-5170-40d9-a7f1-996a87540a6b" path="/var/lib/kubelet/pods/ea256302-5170-40d9-a7f1-996a87540a6b/volumes"
Nov 02 20:28:54.135412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:54.135378    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:28:54.135982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:54.135426    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:28:55.135721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:55.135690    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:28:55.136159 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:55.135746    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:28:56.135644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:56.135607    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:28:56.135855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:56.135653    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:28:57.135546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:57.135512    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:28:57.135834 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:57.135568    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:28:58.140406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:58.140372    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:28:58.140935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:58.140893    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:28:58.141713 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:28:58.141684    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:29:07.712679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:07.712643    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:29:07.713269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:07.712851    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy" containerID="cri-o://eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e" gracePeriod=30
Nov 02 20:29:13.310745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.310722    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:29:13.408457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408376    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408441    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408567    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408604    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408651    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408704    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408737    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:13.408950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408751    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408822    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.408950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408860    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\") pod \"996ed531-bd0a-4227-9082-79327656bc37\" (UID: \"996ed531-bd0a-4227-9082-79327656bc37\") "
Nov 02 20:29:13.409167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.408988    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409262    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409528    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409867    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409892    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409908    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996ed531-bd0a-4227-9082-79327656bc37-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409923    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.410017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.409988    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:13.411188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.411166    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:13.411328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.411207    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:13.411495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.411471    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "kube-api-access-cb8rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:13.411925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.411906    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token" pod "996ed531-bd0a-4227-9082-79327656bc37" (UID: "996ed531-bd0a-4227-9082-79327656bc37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:13.511987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.511944    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.511987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.511983    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.512247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.512000    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8rl\" (UniqueName: \"kubernetes.io/projected/996ed531-bd0a-4227-9082-79327656bc37-kube-api-access-cb8rl\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.512247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.512018    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996ed531-bd0a-4227-9082-79327656bc37-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.512247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.512033    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996ed531-bd0a-4227-9082-79327656bc37-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:13.832026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.831946    5099 generic.go:356] "Generic (PLEG): container finished" podID="996ed531-bd0a-4227-9082-79327656bc37" containerID="eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e" exitCode=0
Nov 02 20:29:13.832026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.831996    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"
Nov 02 20:29:13.832208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.832025    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerDied","Data":"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"}
Nov 02 20:29:13.832208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.832083    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r" event={"ID":"996ed531-bd0a-4227-9082-79327656bc37","Type":"ContainerDied","Data":"acd87c4ce9e4e5335ad0b83835843d013608f7cd26a3533e46759f9a4a857578"}
Nov 02 20:29:13.832208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.832105    5099 scope.go:122] "RemoveContainer" containerID="eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"
Nov 02 20:29:13.849703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.849685    5099 scope.go:122] "RemoveContainer" containerID="ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"
Nov 02 20:29:13.860172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.860152    5099 scope.go:122] "RemoveContainer" containerID="eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"
Nov 02 20:29:13.860458 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:13.860442    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e\": container with ID starting with eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e not found: ID does not exist" containerID="eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"
Nov 02 20:29:13.860512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.860465    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e"} err="failed to get container status \"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e\": rpc error: code = NotFound desc = could not find container \"eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e\": container with ID starting with eeb78b6fcb88dc086c4b351959c0aa5af647efa983f42604b9235f334c06f59e not found: ID does not exist"
Nov 02 20:29:13.860512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.860484    5099 scope.go:122] "RemoveContainer" containerID="ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"
Nov 02 20:29:13.860711 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:13.860694    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a\": container with ID starting with ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a not found: ID does not exist" containerID="ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"
Nov 02 20:29:13.860754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.860717    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a"} err="failed to get container status \"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a\": rpc error: code = NotFound desc = could not find container \"ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a\": container with ID starting with ce82bbb4a0980e83bcd8ad8a7fac8c83de8eeb8f71b7af987f208e2a0f37d96a not found: ID does not exist"
Nov 02 20:29:13.992387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:13.992353    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:29:14.038446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:14.038415    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d5a51f-openshift-default-64476786cc-p4d9r"]
Nov 02 20:29:15.260368 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:15.260322    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996ed531-bd0a-4227-9082-79327656bc37" path="/var/lib/kubelet/pods/996ed531-bd0a-4227-9082-79327656bc37/volumes"
Nov 02 20:29:16.096660 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:16.096621    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:29:17.709049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:17.709012    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:29:17.709586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:17.709218    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy" containerID="cri-o://38ae8d936088a1165f6637433e8f3072c6413bd3870f8f0b5eea4ea645dcbe1a" gracePeriod=30
Nov 02 20:29:17.757822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:17.757749    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:29:17.758050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:17.758017    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy" containerID="cri-o://0997067a5f2e174d4a56ad514e35790cec8e4fed06bb7155f90b066590e8b51d" gracePeriod=30
Nov 02 20:29:18.848464 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.848424    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:29:18.854112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.853983    5099 cpu_manager.go:414] "containerMap: removing container" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:18.854912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.854888    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:18.869819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.855107    5099 cpu_manager.go:414] "containerMap: removing container" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:18.870078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.870051    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:18.870538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.870512    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:18.888010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.885805    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:29:18.888010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:18.886439    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.108861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.083655    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.109220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.109187    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf56t\" (UniqueName: \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.198150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.194904    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1ab501\""
Nov 02 20:29:19.198150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.195059    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1ab501-dockercfg-cltgl\""
Nov 02 20:29:19.214817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.214765    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:29:19.229890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.229848    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.230154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.230135    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf56t\" (UniqueName: \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.231815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.231771    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.266438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.263950    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="996ed531-bd0a-4227-9082-79327656bc37" containerName="istio-proxy"
Nov 02 20:29:19.275760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.270761    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.312452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.310623    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf56t\" (UniqueName: \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\") pod \"payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.319444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.319414    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:29:19.332265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.332235    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq866\" (UniqueName: \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.332438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.332303    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.433627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.433589    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq866\" (UniqueName: \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.433875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.433648    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.434459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.434432    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.487990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.487954    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq866\" (UniqueName: \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\") pod \"payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.606286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.606248    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:19.614694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.614668    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:19.991303 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:29:19.991265    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4fa185_ad29_4705_beec_4dcf27679c0d.slice/crio-dab4879f77f2cd73b4bee8a4248228360df7eb82d3b12516657fc6c586915950 WatchSource:0}: Error finding container dab4879f77f2cd73b4bee8a4248228360df7eb82d3b12516657fc6c586915950: Status 404 returned error can't find the container with id dab4879f77f2cd73b4bee8a4248228360df7eb82d3b12516657fc6c586915950
Nov 02 20:29:19.992225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:19.992196    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:29:20.092606 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:29:20.092576    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fcd0b0_0af2_44b0_aed0_5a959d82623a.slice/crio-8535ba28f7b9ca724c47b8386b8914c44511c8b86863986187375fc195cf8df3 WatchSource:0}: Error finding container 8535ba28f7b9ca724c47b8386b8914c44511c8b86863986187375fc195cf8df3: Status 404 returned error can't find the container with id 8535ba28f7b9ca724c47b8386b8914c44511c8b86863986187375fc195cf8df3
Nov 02 20:29:20.093277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.093255    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:29:20.295706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.295612    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" event={"ID":"52fcd0b0-0af2-44b0-aed0-5a959d82623a","Type":"ContainerStarted","Data":"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"}
Nov 02 20:29:20.295706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.295666    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" event={"ID":"52fcd0b0-0af2-44b0-aed0-5a959d82623a","Type":"ContainerStarted","Data":"8535ba28f7b9ca724c47b8386b8914c44511c8b86863986187375fc195cf8df3"}
Nov 02 20:29:20.295938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.295710    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:20.297346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.297323    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" event={"ID":"bd4fa185-ad29-4705-beec-4dcf27679c0d","Type":"ContainerStarted","Data":"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"}
Nov 02 20:29:20.297479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.297358    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" event={"ID":"bd4fa185-ad29-4705-beec-4dcf27679c0d","Type":"ContainerStarted","Data":"dab4879f77f2cd73b4bee8a4248228360df7eb82d3b12516657fc6c586915950"}
Nov 02 20:29:20.297479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.297390    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:20.339245 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.339216    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:29:20.339470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.339445    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy" containerID="cri-o://30f786f8c99cca94cf5e32e355a95761226b12760d7a394da627e7f7bfc6b986" gracePeriod=30
Nov 02 20:29:20.415027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.414967    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" podStartSLOduration=2.414948205 podStartE2EDuration="2.414948205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:20.412376086 +0000 UTC m=+2249.286646974" watchObservedRunningTime="2026-11-02 20:29:20.414948205 +0000 UTC m=+2249.289219073"
Nov 02 20:29:20.522905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:20.522823    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" podStartSLOduration=2.522804365 podStartE2EDuration="2.522804365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:20.519769732 +0000 UTC m=+2249.394040622" watchObservedRunningTime="2026-11-02 20:29:20.522804365 +0000 UTC m=+2249.397075250"
Nov 02 20:29:23.035722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.035307    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:29:23.035722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.035580    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" containerID="cri-o://f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf" gracePeriod=30
Nov 02 20:29:23.518184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.518148    5099 generic.go:356] "Generic (PLEG): container finished" podID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerID="38ae8d936088a1165f6637433e8f3072c6413bd3870f8f0b5eea4ea645dcbe1a" exitCode=0
Nov 02 20:29:23.518422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.518374    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" event={"ID":"8017ac0c-b3d3-4318-b821-200341fb27ef","Type":"ContainerDied","Data":"38ae8d936088a1165f6637433e8f3072c6413bd3870f8f0b5eea4ea645dcbe1a"}
Nov 02 20:29:23.522777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.522746    5099 generic.go:356] "Generic (PLEG): container finished" podID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerID="0997067a5f2e174d4a56ad514e35790cec8e4fed06bb7155f90b066590e8b51d" exitCode=0
Nov 02 20:29:23.522933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.522862    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" event={"ID":"ce869ac9-9cef-4476-ad89-0def89d9fe65","Type":"ContainerDied","Data":"0997067a5f2e174d4a56ad514e35790cec8e4fed06bb7155f90b066590e8b51d"}
Nov 02 20:29:23.548882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.548847    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:29:23.552453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.552433    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:29:23.672446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672399    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.672602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672485    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.672602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672561    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.672602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672589    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.672768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672659    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.672768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672724    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.672904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672840    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.672904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672882    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.672904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672887    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.673078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.672913    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.673147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673116    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.673291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673262    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.673291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673276    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.673503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673370    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.673503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673480    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:23.673503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673491    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.673736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673538    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.673736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673545    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.673736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673643    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.673736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673709    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.674064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673835    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.674064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673867    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\") pod \"8017ac0c-b3d3-4318-b821-200341fb27ef\" (UID: \"8017ac0c-b3d3-4318-b821-200341fb27ef\") "
Nov 02 20:29:23.674064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673898    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.674064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.673936    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\") pod \"ce869ac9-9cef-4476-ad89-0def89d9fe65\" (UID: \"ce869ac9-9cef-4476-ad89-0def89d9fe65\") "
Nov 02 20:29:23.675273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.675246    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:23.675571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.675542    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.675754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.675736    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.675855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.675754    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.676083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676063    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce869ac9-9cef-4476-ad89-0def89d9fe65-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676089    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676109    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676124    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676139    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676152    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.676454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676430    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "kube-api-access-bdm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:23.676583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676550    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:23.676760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676713    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.676760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.676722    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:23.677641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.677617    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "kube-api-access-rgh4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:23.677862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.677763    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token" pod "8017ac0c-b3d3-4318-b821-200341fb27ef" (UID: "8017ac0c-b3d3-4318-b821-200341fb27ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:23.677937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.677888    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:23.678340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.678323    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo" pod "ce869ac9-9cef-4476-ad89-0def89d9fe65" (UID: "ce869ac9-9cef-4476-ad89-0def89d9fe65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791437    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791469    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791484    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8017ac0c-b3d3-4318-b821-200341fb27ef-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791496    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791510    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791521    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791534    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791547    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdm4f\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-kube-api-access-bdm4f\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791561    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791573    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8017ac0c-b3d3-4318-b821-200341fb27ef-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791587    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:23.793485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:23.791601    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgh4g\" (UniqueName: \"kubernetes.io/projected/ce869ac9-9cef-4476-ad89-0def89d9fe65-kube-api-access-rgh4g\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:24.532359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.532324    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6" event={"ID":"8017ac0c-b3d3-4318-b821-200341fb27ef","Type":"ContainerDied","Data":"8574b12adc2d076e19748ee5e0a48af9c8082b0c50d930310433e90125fc3ff5"}
Nov 02 20:29:24.532729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.532383    5099 scope.go:122] "RemoveContainer" containerID="38ae8d936088a1165f6637433e8f3072c6413bd3870f8f0b5eea4ea645dcbe1a"
Nov 02 20:29:24.532729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.532488    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"
Nov 02 20:29:24.535940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.535916    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"
Nov 02 20:29:24.536095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.535947    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt" event={"ID":"ce869ac9-9cef-4476-ad89-0def89d9fe65","Type":"ContainerDied","Data":"9c4396a8398fb1b3cee6d707dbd0befb5838625a5a2b4a0512e58c8e8836f0b4"}
Nov 02 20:29:24.549607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.549581    5099 scope.go:122] "RemoveContainer" containerID="0997067a5f2e174d4a56ad514e35790cec8e4fed06bb7155f90b066590e8b51d"
Nov 02 20:29:24.743013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.742977    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:29:24.787850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.787743    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-65gbt"]
Nov 02 20:29:24.847496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.847465    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:29:24.854698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:24.854665    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-tsrr6"]
Nov 02 20:29:25.262667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.262631    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" path="/var/lib/kubelet/pods/8017ac0c-b3d3-4318-b821-200341fb27ef/volumes"
Nov 02 20:29:25.263346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.263322    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" path="/var/lib/kubelet/pods/ce869ac9-9cef-4476-ad89-0def89d9fe65/volumes"
Nov 02 20:29:25.553913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.553874    5099 generic.go:356] "Generic (PLEG): container finished" podID="019e565a-a379-4576-b830-48d8f0252824" containerID="30f786f8c99cca94cf5e32e355a95761226b12760d7a394da627e7f7bfc6b986" exitCode=0
Nov 02 20:29:25.554413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.553977    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerDied","Data":"30f786f8c99cca94cf5e32e355a95761226b12760d7a394da627e7f7bfc6b986"}
Nov 02 20:29:25.554413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.554034    5099 scope.go:122] "RemoveContainer" containerID="206164b41ae845d734b404a112bde69dadcf1716f60ea500fea0804dfc0abadd"
Nov 02 20:29:25.645247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.645216    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:29:25.822416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822347    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822394    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822437    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822482    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822694    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822748    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822820    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822829    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:25.822964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822842    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:25.822964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822901    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.822964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.822956    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\") pod \"019e565a-a379-4576-b830-48d8f0252824\" (UID: \"019e565a-a379-4576-b830-48d8f0252824\") "
Nov 02 20:29:25.823201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823049    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:25.823201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823146    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:25.823520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823496    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:25.823848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823825    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.823968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823855    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.823968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823871    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.823968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823886    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.823968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.823901    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/019e565a-a379-4576-b830-48d8f0252824-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.825257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.825230    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:25.825382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.825323    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:25.825382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.825354    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:25.825624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.825606    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd" pod "019e565a-a379-4576-b830-48d8f0252824" (UID: "019e565a-a379-4576-b830-48d8f0252824"). InnerVolumeSpecName "kube-api-access-q46pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:25.925318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.925265    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/019e565a-a379-4576-b830-48d8f0252824-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.925318 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.925319    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.925536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.925334    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46pd\" (UniqueName: \"kubernetes.io/projected/019e565a-a379-4576-b830-48d8f0252824-kube-api-access-q46pd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:25.925536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:25.925346    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/019e565a-a379-4576-b830-48d8f0252824-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:26.594803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:26.593717    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln" event={"ID":"019e565a-a379-4576-b830-48d8f0252824","Type":"ContainerDied","Data":"3fe506c1787e8b119eab86518c88b693ca88e7080db5b2de7a67d5671f86db37"}
Nov 02 20:29:26.594803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:26.593840    5099 scope.go:122] "RemoveContainer" containerID="30f786f8c99cca94cf5e32e355a95761226b12760d7a394da627e7f7bfc6b986"
Nov 02 20:29:26.594803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:26.594015    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"
Nov 02 20:29:26.801854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:26.801820    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:29:26.849039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:26.848959    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-w6pln"]
Nov 02 20:29:27.263475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:27.263444    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="019e565a-a379-4576-b830-48d8f0252824" path="/var/lib/kubelet/pods/019e565a-a379-4576-b830-48d8f0252824/volumes"
Nov 02 20:29:28.142164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.142126    5099 patch_prober.go:29] interesting pod/e2e-api-a-abc937-openshift-default-8884f867f-stldm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 20:29:28.142576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.142203    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 20:29:28.402264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.402238    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:29:28.570236 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570200    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570310    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570458    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570491    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570518    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:28.570732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570597    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570620    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570645    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.570732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.570689    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\") pod \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\" (UID: \"864dc561-84ac-4b6e-9151-7fe3b0f21f2e\") "
Nov 02 20:29:28.571385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.571345    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.572146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.572044    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:28.572301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.572274    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:28.572568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.572542    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:28.572818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.572795    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:28.574872 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.574842    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:28.574987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.574900    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "kube-api-access-kvkkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:28.575196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.575172    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:28.576096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.576075    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy" pod "864dc561-84ac-4b6e-9151-7fe3b0f21f2e" (UID: "864dc561-84ac-4b6e-9151-7fe3b0f21f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:28.619092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.619012    5099 generic.go:356] "Generic (PLEG): container finished" podID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerID="f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf" exitCode=0
Nov 02 20:29:28.619092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.619069    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"
Nov 02 20:29:28.619254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.619088    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerDied","Data":"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"}
Nov 02 20:29:28.619254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.619150    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm" event={"ID":"864dc561-84ac-4b6e-9151-7fe3b0f21f2e","Type":"ContainerDied","Data":"bb7dde157a985f748215d183583effc694fd6d857acf7ee12136f79faca788dd"}
Nov 02 20:29:28.619254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.619174    5099 scope.go:122] "RemoveContainer" containerID="f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"
Nov 02 20:29:28.631289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.631268    5099 scope.go:122] "RemoveContainer" containerID="603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"
Nov 02 20:29:28.644435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.644405    5099 scope.go:122] "RemoveContainer" containerID="f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"
Nov 02 20:29:28.644737 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:28.644712    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf\": container with ID starting with f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf not found: ID does not exist" containerID="f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"
Nov 02 20:29:28.644894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.644748    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf"} err="failed to get container status \"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf\": rpc error: code = NotFound desc = could not find container \"f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf\": container with ID starting with f6ebe25a2ecd7fe0b25727914852b84685fa8eaee440e7d00d53f6d72a1cb7cf not found: ID does not exist"
Nov 02 20:29:28.644894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.644775    5099 scope.go:122] "RemoveContainer" containerID="603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"
Nov 02 20:29:28.645495 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:28.645466    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6\": container with ID starting with 603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6 not found: ID does not exist" containerID="603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"
Nov 02 20:29:28.645589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.645502    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6"} err="failed to get container status \"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6\": rpc error: code = NotFound desc = could not find container \"603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6\": container with ID starting with 603b8bbe95da920b9db3e4adbdc33a9481fbc1d1e8308afa68c52f18cb55e3a6 not found: ID does not exist"
Nov 02 20:29:28.672656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672626    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672661    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672678    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672692    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvkkz\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-kube-api-access-kvkkz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672708    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672724    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672737    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.672777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.672748    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864dc561-84ac-4b6e-9151-7fe3b0f21f2e-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:28.780009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.779975    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:29:28.786576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:28.786546    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-abc937-openshift-default-8884f867f-stldm"]
Nov 02 20:29:29.262003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:29.261971    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" path="/var/lib/kubelet/pods/864dc561-84ac-4b6e-9151-7fe3b0f21f2e/volumes"
Nov 02 20:29:31.327092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:31.327040    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:29:31.346981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:31.346952    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:29:35.562566 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.562526    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:29:35.563561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563533    5099 cpu_manager.go:414] "containerMap: removing container" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:35.563561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563556    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563574    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563580    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563594    5099 cpu_manager.go:414] "containerMap: removing container" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563598    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563610    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563616    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563626    5099 cpu_manager.go:414] "containerMap: removing container" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563630    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563762    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce869ac9-9cef-4476-ad89-0def89d9fe65" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563772    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:35.563769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563793    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8017ac0c-b3d3-4318-b821-200341fb27ef" containerName="istio-proxy"
Nov 02 20:29:35.564443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.563800    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:35.569465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.569427    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.630401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.630363    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:29:35.669029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.668989    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62d287-openshift-default-dockercfg-7kr7s\""
Nov 02 20:29:35.706205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706169    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706382    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706442    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706559    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706626    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706778 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706765    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706824    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82vn\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.706958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706936    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.707018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.706986    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.811611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.811571    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.811843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.811652    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.811843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.811691    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.812946 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.812885    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814396    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814524    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814585    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82vn\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814647    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814685    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.814817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.814726    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.815246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.815220    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.815390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.815365    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.815649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.815629    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.815828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.815704    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.825029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.825000    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.825253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.825232    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.868873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.868842    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.869028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.869008    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82vn\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\") pod \"e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") " pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:35.885084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:35.885061    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:36.545459 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.544240    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:29:36.553733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.553693    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:36.554000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.553979    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:36.554136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.554119    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:36.686488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.686451    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerStarted","Data":"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"}
Nov 02 20:29:36.687036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.686515    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerStarted","Data":"eb733567ba25cb6bae41dfffc24815fed07b2007a792595dce7b385a13e446b2"}
Nov 02 20:29:36.797264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.797127    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podStartSLOduration=1.797110835 podStartE2EDuration="1.797110835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:36.79666505 +0000 UTC m=+2265.670935963" watchObservedRunningTime="2026-11-02 20:29:36.797110835 +0000 UTC m=+2265.671381724"
Nov 02 20:29:36.885550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:36.885516    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:37.470375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:37.470339    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"]
Nov 02 20:29:37.475982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:37.475952    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dcb03-c72xv"]
Nov 02 20:29:37.892479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:37.892444    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:38.705669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:38.705638    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:38.706551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:38.706526    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:29:39.261397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:39.261357    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ef1507-c70f-49aa-af51-7bb5bbffa55d" path="/var/lib/kubelet/pods/60ef1507-c70f-49aa-af51-7bb5bbffa55d/volumes"
Nov 02 20:29:40.211762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.211727    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:29:40.211990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.211963    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy" containerID="cri-o://e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a" gracePeriod=30
Nov 02 20:29:40.259234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.259200    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:29:40.259465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.259439    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy" containerID="cri-o://1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74" gracePeriod=30
Nov 02 20:29:40.265546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.265516    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:29:40.266002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:40.265681    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy" containerID="cri-o://07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea" gracePeriod=30
Nov 02 20:29:43.857226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:43.850888    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"]
Nov 02 20:29:43.922238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:43.920418    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-81e58f-tchcl"]
Nov 02 20:29:44.880841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:44.880806    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:29:44.881299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:44.881021    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy" containerID="cri-o://008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282" gracePeriod=30
Nov 02 20:29:45.267092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.267061    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0850b6f7-f3ae-41dd-8824-aaacabc84bef" path="/var/lib/kubelet/pods/0850b6f7-f3ae-41dd-8824-aaacabc84bef/volumes"
Nov 02 20:29:45.592151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.592065    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:29:45.653674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.653632    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:29:45.678058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.678031    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:29:45.687237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687209    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687258    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.687374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687294    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687343    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687382    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687419    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.687547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687450    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.687547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687518    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687569    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.687763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687698    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.687763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687713    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.687938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687760    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:45.687938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687767    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.687938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687824    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687859    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.687938 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687923    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687957    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687963    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.687960    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688059    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\") pod \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\" (UID: \"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b\") "
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688091    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688108    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688141    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.688189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688211    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688238    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688293    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688348    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688439    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688524    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688554    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688584    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.688680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688647    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.689156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688707    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\") pod \"4311b6c0-0384-40b5-b78f-9edee65e8051\" (UID: \"4311b6c0-0384-40b5-b78f-9edee65e8051\") "
Nov 02 20:29:45.689156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.688774    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\") pod \"24af1733-2694-4a58-9a8c-3758912188bd\" (UID: \"24af1733-2694-4a58-9a8c-3758912188bd\") "
Nov 02 20:29:45.689728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689702    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689733    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689750    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689765    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689795    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689809    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689822    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24af1733-2694-4a58-9a8c-3758912188bd-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.689859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689836    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.690237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.689973    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.690237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.690044    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.690237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.690199    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.690675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.690653    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.690898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.690843    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "kube-api-access-jrjbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.690994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.690946    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "kube-api-access-6cx88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.691170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.691153    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.692387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692367    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "kube-api-access-h6hvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.692589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692572    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.692715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692632    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:45.692805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692755    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.692871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692838    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.693013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692991    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.693089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.692999    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:45.693089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.693018    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:45.693212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.693137    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.693267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.693253    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy" pod "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" (UID: "42ee9d4e-6a47-4b42-87a1-0d6c0f26341b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:45.693663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.693639    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo" pod "4311b6c0-0384-40b5-b78f-9edee65e8051" (UID: "4311b6c0-0384-40b5-b78f-9edee65e8051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:45.693965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.693948    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token" pod "24af1733-2694-4a58-9a8c-3758912188bd" (UID: "24af1733-2694-4a58-9a8c-3758912188bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791240    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791273    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791289    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cx88\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-kube-api-access-6cx88\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791302    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791314    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjbb\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-kube-api-access-jrjbb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791327    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791340    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791352    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791364    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24af1733-2694-4a58-9a8c-3758912188bd-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791376    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791388    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24af1733-2694-4a58-9a8c-3758912188bd-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791400    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4311b6c0-0384-40b5-b78f-9edee65e8051-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791412    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791423    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hvs\" (UniqueName: \"kubernetes.io/projected/4311b6c0-0384-40b5-b78f-9edee65e8051-kube-api-access-h6hvs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791436    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791447    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791459    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791471    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4311b6c0-0384-40b5-b78f-9edee65e8051-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.791705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.791487    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24af1733-2694-4a58-9a8c-3758912188bd-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:45.916759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.916723    5099 generic.go:356] "Generic (PLEG): container finished" podID="24af1733-2694-4a58-9a8c-3758912188bd" containerID="1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74" exitCode=0
Nov 02 20:29:45.917197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.916812    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" event={"ID":"24af1733-2694-4a58-9a8c-3758912188bd","Type":"ContainerDied","Data":"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"}
Nov 02 20:29:45.917197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.916871    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc" event={"ID":"24af1733-2694-4a58-9a8c-3758912188bd","Type":"ContainerDied","Data":"ca286224b47926a23341dac06739c6aa3d4e2ec32d9959c8d75a8293414557d6"}
Nov 02 20:29:45.917197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.916822    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"
Nov 02 20:29:45.917197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.916897    5099 scope.go:122] "RemoveContainer" containerID="1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"
Nov 02 20:29:45.919330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.918906    5099 generic.go:356] "Generic (PLEG): container finished" podID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerID="07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea" exitCode=0
Nov 02 20:29:45.919330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.919064    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" event={"ID":"4311b6c0-0384-40b5-b78f-9edee65e8051","Type":"ContainerDied","Data":"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"}
Nov 02 20:29:45.919330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.919107    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb" event={"ID":"4311b6c0-0384-40b5-b78f-9edee65e8051","Type":"ContainerDied","Data":"f3016b69ebee7b2f89a3dd5cd6443eb30a013f46a3feed4f24908c1761ddda80"}
Nov 02 20:29:45.920164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.919760    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"
Nov 02 20:29:45.920706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.920675    5099 generic.go:356] "Generic (PLEG): container finished" podID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerID="e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a" exitCode=0
Nov 02 20:29:45.920846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.920773    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" event={"ID":"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b","Type":"ContainerDied","Data":"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"}
Nov 02 20:29:45.920846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.920840    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4" event={"ID":"42ee9d4e-6a47-4b42-87a1-0d6c0f26341b","Type":"ContainerDied","Data":"14049544758fbcabb3ef323f0295e78ea2698313072f22231d4bee0207f77469"}
Nov 02 20:29:45.920984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.920924    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"
Nov 02 20:29:45.947667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.947638    5099 scope.go:122] "RemoveContainer" containerID="1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"
Nov 02 20:29:45.948029 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:45.948007    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74\": container with ID starting with 1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74 not found: ID does not exist" containerID="1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"
Nov 02 20:29:45.948126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.948043    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74"} err="failed to get container status \"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74\": rpc error: code = NotFound desc = could not find container \"1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74\": container with ID starting with 1f6681d71689881b82a1544b153e2dc6e14c9f6f841d9e3b2fead27e31f63d74 not found: ID does not exist"
Nov 02 20:29:45.948126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.948068    5099 scope.go:122] "RemoveContainer" containerID="07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"
Nov 02 20:29:45.961715 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.961692    5099 scope.go:122] "RemoveContainer" containerID="07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"
Nov 02 20:29:45.962134 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:45.962105    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea\": container with ID starting with 07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea not found: ID does not exist" containerID="07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"
Nov 02 20:29:45.962238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.962147    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea"} err="failed to get container status \"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea\": rpc error: code = NotFound desc = could not find container \"07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea\": container with ID starting with 07be323241354aff1026baaaf528cfb2dd957975b43b8d7c7da00aa9982c1eea not found: ID does not exist"
Nov 02 20:29:45.962238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.962171    5099 scope.go:122] "RemoveContainer" containerID="e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"
Nov 02 20:29:45.974917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.974873    5099 scope.go:122] "RemoveContainer" containerID="e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"
Nov 02 20:29:45.975227 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:45.975205    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a\": container with ID starting with e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a not found: ID does not exist" containerID="e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"
Nov 02 20:29:45.975320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:45.975232    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a"} err="failed to get container status \"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a\": rpc error: code = NotFound desc = could not find container \"e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a\": container with ID starting with e6877431c1549a1f671516df82ef393f1f8db2c9347cfbc552a7d4102ed87b4a not found: ID does not exist"
Nov 02 20:29:46.080937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.080854    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:29:46.085219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.085193    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-fpndb"]
Nov 02 20:29:46.105116 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:46.105084    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:29:46.180406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.180370    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:29:46.184701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.184662    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-gmgt4"]
Nov 02 20:29:46.284990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.284961    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:29:46.289036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:46.289010    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-htsgc"]
Nov 02 20:29:47.259605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:47.259569    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24af1733-2694-4a58-9a8c-3758912188bd" path="/var/lib/kubelet/pods/24af1733-2694-4a58-9a8c-3758912188bd/volumes"
Nov 02 20:29:47.260164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:47.260147    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" path="/var/lib/kubelet/pods/42ee9d4e-6a47-4b42-87a1-0d6c0f26341b/volumes"
Nov 02 20:29:47.260636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:47.260622    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" path="/var/lib/kubelet/pods/4311b6c0-0384-40b5-b78f-9edee65e8051/volumes"
Nov 02 20:29:49.941922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:49.941889    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:29:49.942555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:49.942114    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" containerID="cri-o://08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568" gracePeriod=30
Nov 02 20:29:50.416421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.416395    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:29:50.559578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.559490    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.559930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.559551    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560068    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560174    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560231    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560307    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560378    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560425    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\") pod \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\" (UID: \"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c\") "
Nov 02 20:29:50.560642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560571    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:50.561033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.560899    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:50.561167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561107    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:50.561285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561269    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:50.561443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561419    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.561584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561446    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.561584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561460    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.561703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.561578    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:50.567628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.562338    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:50.571217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.571192    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:50.571569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.571546    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:50.572178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.572142    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82" pod "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" (UID: "54c1eb63-b8e9-4ce0-9f29-1e758e742c5c"). InnerVolumeSpecName "kube-api-access-xtd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:50.663255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663220    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.663255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663254    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.663502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663270    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.663502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663283    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.663502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663295    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.663502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.663308    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtd82\" (UniqueName: \"kubernetes.io/projected/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c-kube-api-access-xtd82\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:50.968151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.968115    5099 generic.go:356] "Generic (PLEG): container finished" podID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerID="008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282" exitCode=0
Nov 02 20:29:50.968597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.968170    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"
Nov 02 20:29:50.968597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.968192    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerDied","Data":"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"}
Nov 02 20:29:50.968597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.968248    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng" event={"ID":"54c1eb63-b8e9-4ce0-9f29-1e758e742c5c","Type":"ContainerDied","Data":"9cef7ef0a6c5655722af6626a51f1fc6699832bd4d5ebd1ac9cd813115164a92"}
Nov 02 20:29:50.968597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.968272    5099 scope.go:122] "RemoveContainer" containerID="008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"
Nov 02 20:29:50.981675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.981656    5099 scope.go:122] "RemoveContainer" containerID="9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"
Nov 02 20:29:50.992265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.992237    5099 scope.go:122] "RemoveContainer" containerID="008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"
Nov 02 20:29:50.992567 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:50.992540    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282\": container with ID starting with 008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282 not found: ID does not exist" containerID="008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"
Nov 02 20:29:50.992655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.992571    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282"} err="failed to get container status \"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282\": rpc error: code = NotFound desc = could not find container \"008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282\": container with ID starting with 008af23210b8f1ea53eeb937d0a2a08919f6eae6413181590bd5f1b93acd4282 not found: ID does not exist"
Nov 02 20:29:50.992655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.992590    5099 scope.go:122] "RemoveContainer" containerID="9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"
Nov 02 20:29:50.992883 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:50.992862    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc\": container with ID starting with 9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc not found: ID does not exist" containerID="9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"
Nov 02 20:29:50.992956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:50.992890    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc"} err="failed to get container status \"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc\": rpc error: code = NotFound desc = could not find container \"9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc\": container with ID starting with 9672fbcb7882f8b5f9660f2ac7f6ef1f6edea02ff8ad288127aca08c5ced75dc not found: ID does not exist"
Nov 02 20:29:51.128635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:51.128599    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:29:51.174092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:51.174056    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-x9qng"]
Nov 02 20:29:51.268584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:51.268502    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" path="/var/lib/kubelet/pods/54c1eb63-b8e9-4ce0-9f29-1e758e742c5c/volumes"
Nov 02 20:29:52.509578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.509545    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:29:52.510453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510430    5099 cpu_manager.go:414] "containerMap: removing container" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:52.510453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510450    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510466    5099 cpu_manager.go:414] "containerMap: removing container" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510472    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510482    5099 cpu_manager.go:414] "containerMap: removing container" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510488    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510512    5099 cpu_manager.go:414] "containerMap: removing container" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510517    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510529    5099 cpu_manager.go:414] "containerMap: removing container" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510534    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510543    5099 cpu_manager.go:414] "containerMap: removing container" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.510606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510548    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510681    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="019e565a-a379-4576-b830-48d8f0252824" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510692    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="864dc561-84ac-4b6e-9151-7fe3b0f21f2e" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510701    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="24af1733-2694-4a58-9a8c-3758912188bd" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510708    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="4311b6c0-0384-40b5-b78f-9edee65e8051" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510717    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ee9d4e-6a47-4b42-87a1-0d6c0f26341b" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510724    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.511020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.510731    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c1eb63-b8e9-4ce0-9f29-1e758e742c5c" containerName="istio-proxy"
Nov 02 20:29:52.516312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.516292    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593069    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593305    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593359    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjj2t\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593503 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593483    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593531    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593637    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593720 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593696    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593797    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.593870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.593856    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.609373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.609340    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:29:52.612884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.612856    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-65f28f-openshift-default-dockercfg-whrxx\""
Nov 02 20:29:52.713858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.707230    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.744817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.736516    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjj2t\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.744817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.736631    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.744817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.734546    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.744817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.736720    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.754824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.753884    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.754824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.754007    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.754824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.754071    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.754824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.754341    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.754824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.754438    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.782252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.782163    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.782437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.782410    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.783116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.783082    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.804417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.784570    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.807008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.806977    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.848530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.823134    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.867196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.864150    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:52.867196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:52.865728    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjj2t\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\") pod \"e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") " pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:53.131761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.131725    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:53.565069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.564965    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:29:53.600153 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:29:53.600120    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b58d473_99cb_4c46_bb41_77c27a057533.slice/crio-15b53bc7459569c5ac4627e0e93bc52147e19ff7f7e0487cfce84af82a0922bb WatchSource:0}: Error finding container 15b53bc7459569c5ac4627e0e93bc52147e19ff7f7e0487cfce84af82a0922bb: Status 404 returned error can't find the container with id 15b53bc7459569c5ac4627e0e93bc52147e19ff7f7e0487cfce84af82a0922bb
Nov 02 20:29:53.603033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.602998    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:53.603116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.603074    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:53.603116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.603104    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:29:53.972797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.972755    5099 patch_prober.go:29] interesting pod/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:29:53.972982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.972853    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:29:53.997848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.997815    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerStarted","Data":"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"}
Nov 02 20:29:53.998027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:53.997877    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerStarted","Data":"15b53bc7459569c5ac4627e0e93bc52147e19ff7f7e0487cfce84af82a0922bb"}
Nov 02 20:29:54.117076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:54.117019    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podStartSLOduration=2.11700297 podStartE2EDuration="2.11700297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:54.113115522 +0000 UTC m=+2282.987386431" watchObservedRunningTime="2026-11-02 20:29:54.11700297 +0000 UTC m=+2282.991273936"
Nov 02 20:29:54.132313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:54.132279    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:54.138114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:54.138082    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:55.007027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.006994    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:55.008178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.008156    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:29:55.185023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.184899    5099 scope.go:122] "RemoveContainer" containerID="72d0baad3729ec129a1d25225df841e41eb1f7788bf76ef4698c40854d586541"
Nov 02 20:29:55.297126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.297102    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:29:55.339404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339372    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339458    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339539    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339601    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339623    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:55.339755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339678    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339726    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:55.339755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339746    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339811    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339880    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.339963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339926    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\") pod \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\" (UID: \"e793033c-8ad8-4d36-87cc-0de4a0a7eda5\") "
Nov 02 20:29:55.340115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.339981    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:55.340115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340060    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:29:55.340212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340169    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:55.341012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340909    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.341012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340937    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.341012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340951    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.341012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340966    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.341012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.340981    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.342499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.342477    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:29:55.342595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.342565    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "kube-api-access-stc7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:55.342851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.342825    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:29:55.343327 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.343309    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo" pod "e793033c-8ad8-4d36-87cc-0de4a0a7eda5" (UID: "e793033c-8ad8-4d36-87cc-0de4a0a7eda5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:29:55.442030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.441998    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.442030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.442028    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.442250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.442041    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:55.442250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:55.442054    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stc7d\" (UniqueName: \"kubernetes.io/projected/e793033c-8ad8-4d36-87cc-0de4a0a7eda5-kube-api-access-stc7d\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:29:56.017252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.017211    5099 generic.go:356] "Generic (PLEG): container finished" podID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerID="08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568" exitCode=0
Nov 02 20:29:56.017766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.017296    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"
Nov 02 20:29:56.017766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.017295    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerDied","Data":"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"}
Nov 02 20:29:56.017766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.017445    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l" event={"ID":"e793033c-8ad8-4d36-87cc-0de4a0a7eda5","Type":"ContainerDied","Data":"4176a0b8e445225e7bf879a8194098a6e83ad2f6defdd4183d3d03a216174ecb"}
Nov 02 20:29:56.017766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.017475    5099 scope.go:122] "RemoveContainer" containerID="08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"
Nov 02 20:29:56.030071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.030052    5099 scope.go:122] "RemoveContainer" containerID="37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"
Nov 02 20:29:56.041817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.041764    5099 scope.go:122] "RemoveContainer" containerID="08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"
Nov 02 20:29:56.042127 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:56.042100    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568\": container with ID starting with 08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568 not found: ID does not exist" containerID="08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"
Nov 02 20:29:56.042210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.042137    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568"} err="failed to get container status \"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568\": rpc error: code = NotFound desc = could not find container \"08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568\": container with ID starting with 08eb7a99b50211699bed01157e52a8a86934413c323870c7237b18617cd73568 not found: ID does not exist"
Nov 02 20:29:56.042210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.042155    5099 scope.go:122] "RemoveContainer" containerID="37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"
Nov 02 20:29:56.042426 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:29:56.042403    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a\": container with ID starting with 37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a not found: ID does not exist" containerID="37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"
Nov 02 20:29:56.042487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.042435    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a"} err="failed to get container status \"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a\": rpc error: code = NotFound desc = could not find container \"37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a\": container with ID starting with 37eafe0661d47b1790fa9c395e641a7c4b26e11f64d92a88a005541bbabc4f1a not found: ID does not exist"
Nov 02 20:29:56.177508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.177472    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:29:56.222275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:56.222242    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-81e58f-openshift-default-6fc9c5bf5-twn6l"]
Nov 02 20:29:57.179262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:57.179229    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:29:57.264363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:29:57.264274    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" path="/var/lib/kubelet/pods/e793033c-8ad8-4d36-87cc-0de4a0a7eda5/volumes"
Nov 02 20:30:00.166004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.165914    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:00.167276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167253    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.167353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167279    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.167353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167301    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.167353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167309    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.167525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167512    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.167558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.167528    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e793033c-8ad8-4d36-87cc-0de4a0a7eda5" containerName="istio-proxy"
Nov 02 20:30:00.174202 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.174177    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.176579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.176550    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:00.187270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.187244    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.192746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.192716    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:00.201449 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.201429    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.212896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.212871    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:00.217605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.217534    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.219271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.219246    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:00.224893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.224872    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:00.225011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.224988    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.229948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.229929    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:00.230061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.230038    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.244502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.244477    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.252442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252413    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b97p\" (UniqueName: \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.252575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252470    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.252575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252550    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwmk\" (UniqueName: \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.252575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252575    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.252768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252698    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmcbn\" (UniqueName: \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.252768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252758    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpsk\" (UniqueName: \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.252933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252862    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.252996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.252964    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.258418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.258395    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:00.302824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.299928    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:00.306901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.306871    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:00.313253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.313222    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:00.324210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.324186    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:30:00.349575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.349357    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:00.363611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363574    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nnv\" (UniqueName: \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.363813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363634    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpsk\" (UniqueName: \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.363813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363680    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.363813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363741    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.364029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363822    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.364029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363938    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b97p\" (UniqueName: \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.364029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.363986    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.364029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364019    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.364255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364103    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwmk\" (UniqueName: \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.364255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364140    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.364255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364187    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnc2j\" (UniqueName: \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.364255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364223    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.364462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364292    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrd9\" (UniqueName: \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.364462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364360    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmcbn\" (UniqueName: \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.366549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.364714    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.366549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.365438    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.366549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.365486    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.366549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.366052    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.371574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.370979    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:00.399387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.398429    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:00.421085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.420978    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmcbn\" (UniqueName: \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\") pod \"maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.421489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.421369    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpsk\" (UniqueName: \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.425492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.425468    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b97p\" (UniqueName: \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.425596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.425482    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwmk\" (UniqueName: \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.466486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466447    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.466688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466617    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.466736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466703    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnc2j\" (UniqueName: \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.466793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466738    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.466846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466813    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrd9\" (UniqueName: \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.467243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.466916    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nnv\" (UniqueName: \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.467369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.467266    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.467369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.467328    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.467643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.467619    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.505358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.505330    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:00.521876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.521840    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnc2j\" (UniqueName: \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\") pod \"maas-api-key-cleanup-29894190-zvzf9\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.522527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.522507    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrd9\" (UniqueName: \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.522699 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.522673    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nnv\" (UniqueName: \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\") pod \"maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.526313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.526293    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:00.539732 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.539514    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:00.548326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.548138    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:00.563851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.563097    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:00.574053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.574012    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:00.587413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.585614    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:00.906842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:00.906808    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:00.908174 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:30:00.908142    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ceb221_dbf3_4dac_a0e6_fbc437103f99.slice/crio-8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1 WatchSource:0}: Error finding container 8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1: Status 404 returned error can't find the container with id 8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1
Nov 02 20:30:01.007884 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:30:01.007857    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38efc7ac_860b_4800_9a82_fb84db5f66fe.slice/crio-888e2742f22d2e6ca2f08db8badcffb06e930b03d3088cc9a51b64aef977f2df WatchSource:0}: Error finding container 888e2742f22d2e6ca2f08db8badcffb06e930b03d3088cc9a51b64aef977f2df: Status 404 returned error can't find the container with id 888e2742f22d2e6ca2f08db8badcffb06e930b03d3088cc9a51b64aef977f2df
Nov 02 20:30:01.009517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.009489    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:01.073589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.073558    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:01.073850 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:30:01.073822    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4255fef_8f15_43d4_a8c3_ea8c18519fec.slice/crio-906d191ed8f0fd41b8fbe69266a4dde7da91b62e618083a3e7a2fc7b84400dd0 WatchSource:0}: Error finding container 906d191ed8f0fd41b8fbe69266a4dde7da91b62e618083a3e7a2fc7b84400dd0: Status 404 returned error can't find the container with id 906d191ed8f0fd41b8fbe69266a4dde7da91b62e618083a3e7a2fc7b84400dd0
Nov 02 20:30:01.075119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.075010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerStarted","Data":"888e2742f22d2e6ca2f08db8badcffb06e930b03d3088cc9a51b64aef977f2df"}
Nov 02 20:30:01.080617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.080595    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerStarted","Data":"8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1"}
Nov 02 20:30:01.106928 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.106904    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:01.115052 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:30:01.115022    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3476cb95_0386_45f9_b490_e0c5f7fbabb7.slice/crio-0068bbd8ec2a3e0395ed508bd7f4227f7351c7b06b59a74c31ebf14f636e5afe WatchSource:0}: Error finding container 0068bbd8ec2a3e0395ed508bd7f4227f7351c7b06b59a74c31ebf14f636e5afe: Status 404 returned error can't find the container with id 0068bbd8ec2a3e0395ed508bd7f4227f7351c7b06b59a74c31ebf14f636e5afe
Nov 02 20:30:01.115377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.115356    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:01.280497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.280472    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:01.280863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:01.280517    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:02.091914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.091878    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerStarted","Data":"09b2035195c4ae8315d3cd67406e0a6a1ef118683354aa9b84fba3aeb551aeee"}
Nov 02 20:30:02.093656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.093634    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerStarted","Data":"561d3515dc4b5875d0f7ebb4782a807156e78f27b692cd493df2980bca51fc68"}
Nov 02 20:30:02.093773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.093677    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerStarted","Data":"147137ea0fbe60ee6430bba9aed3225d01b5023c857d6040ed2005b34b708bec"}
Nov 02 20:30:02.095361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.095331    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerStarted","Data":"e7359805e735d57a1a190c3fe88cf8f35f732a37ac382072c786e52f96aa8a14"}
Nov 02 20:30:02.095473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.095375    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerStarted","Data":"3d230d743898585ff6f3f13a528f53de70eae42088c29d50d8a80409fd56f01a"}
Nov 02 20:30:02.096945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.096925    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerStarted","Data":"fb995759a54572ee60fae4ad69f194998992f1108956091ff09bc75db43814eb"}
Nov 02 20:30:02.097028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.096964    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerStarted","Data":"0068bbd8ec2a3e0395ed508bd7f4227f7351c7b06b59a74c31ebf14f636e5afe"}
Nov 02 20:30:02.098634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.098571    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerStarted","Data":"49d7bdb5f410dc15f0ae66ae9df6173582fc1fc64523de8b94b2a6f66999f7ec"}
Nov 02 20:30:02.098634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.098608    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerStarted","Data":"63415f7b41e55678feb2d6b4d7e26f9b78e0803c5fa8539ce3dac286bd69010d"}
Nov 02 20:30:02.100123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.100101    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerStarted","Data":"2f1ac5c5f3ca9023e7647caed8019c33cb1368d372b5445d4c71deb8def14b15"}
Nov 02 20:30:02.101599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.101580    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerStarted","Data":"77a832289b96d37e1f2617fc5b94f92d9d290ee08bf9ef6647f21ac30ab9baea"}
Nov 02 20:30:02.101688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.101612    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerStarted","Data":"906d191ed8f0fd41b8fbe69266a4dde7da91b62e618083a3e7a2fc7b84400dd0"}
Nov 02 20:30:02.179966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.179773    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" podUID="6df451b8-1cfd-4b23-b289-30257a582317" containerName="payload-processing" containerID="cri-o://cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99" gracePeriod=25
Nov 02 20:30:02.205515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.205429    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" podStartSLOduration=2.205408735 podStartE2EDuration="2.205408735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.199268361 +0000 UTC m=+2291.073539249" watchObservedRunningTime="2026-11-02 20:30:02.205408735 +0000 UTC m=+2291.079679623"
Nov 02 20:30:02.308600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.308552    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" podStartSLOduration=2.30854008 podStartE2EDuration="2.30854008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.305381888 +0000 UTC m=+2291.179652776" watchObservedRunningTime="2026-11-02 20:30:02.30854008 +0000 UTC m=+2291.182810967"
Nov 02 20:30:02.417117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.417060    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" podStartSLOduration=2.417042237 podStartE2EDuration="2.417042237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.411955465 +0000 UTC m=+2291.286226352" watchObservedRunningTime="2026-11-02 20:30:02.417042237 +0000 UTC m=+2291.291313124"
Nov 02 20:30:02.522541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.522490    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" podStartSLOduration=2.5224769179999997 podStartE2EDuration="2.522476918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.518037431 +0000 UTC m=+2291.392308320" watchObservedRunningTime="2026-11-02 20:30:02.522476918 +0000 UTC m=+2291.396747805"
Nov 02 20:30:02.626423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.626334    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" podStartSLOduration=2.626320197 podStartE2EDuration="2.626320197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.624736895 +0000 UTC m=+2291.499007805" watchObservedRunningTime="2026-11-02 20:30:02.626320197 +0000 UTC m=+2291.500591086"
Nov 02 20:30:02.732442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.732397    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" podStartSLOduration=2.732382904 podStartE2EDuration="2.732382904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.731387613 +0000 UTC m=+2291.605658499" watchObservedRunningTime="2026-11-02 20:30:02.732382904 +0000 UTC m=+2291.606653840"
Nov 02 20:30:02.733789 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.733759    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:30:02.819108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.819074    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\" (UniqueName: \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\") pod \"6df451b8-1cfd-4b23-b289-30257a582317\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") "
Nov 02 20:30:02.819341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.819323    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\") pod \"6df451b8-1cfd-4b23-b289-30257a582317\" (UID: \"6df451b8-1cfd-4b23-b289-30257a582317\") "
Nov 02 20:30:02.819728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.819700    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume" pod "6df451b8-1cfd-4b23-b289-30257a582317" (UID: "6df451b8-1cfd-4b23-b289-30257a582317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:02.820003 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.819976    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df451b8-1cfd-4b23-b289-30257a582317-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:02.822185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.822150    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm" pod "6df451b8-1cfd-4b23-b289-30257a582317" (UID: "6df451b8-1cfd-4b23-b289-30257a582317"). InnerVolumeSpecName "kube-api-access-mt4nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:02.841104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.841055    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" podStartSLOduration=2.841042784 podStartE2EDuration="2.841042784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.836994121 +0000 UTC m=+2291.711265010" watchObservedRunningTime="2026-11-02 20:30:02.841042784 +0000 UTC m=+2291.715313671"
Nov 02 20:30:02.921884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:02.921776    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt4nm\" (UniqueName: \"kubernetes.io/projected/6df451b8-1cfd-4b23-b289-30257a582317-kube-api-access-mt4nm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:03.111902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.111868    5099 generic.go:356] "Generic (PLEG): container finished" podID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerID="09b2035195c4ae8315d3cd67406e0a6a1ef118683354aa9b84fba3aeb551aeee" exitCode=60
Nov 02 20:30:03.112089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.111948    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerDied","Data":"09b2035195c4ae8315d3cd67406e0a6a1ef118683354aa9b84fba3aeb551aeee"}
Nov 02 20:30:03.112375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.112349    5099 scope.go:122] "RemoveContainer" containerID="09b2035195c4ae8315d3cd67406e0a6a1ef118683354aa9b84fba3aeb551aeee"
Nov 02 20:30:03.113961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.113940    5099 generic.go:356] "Generic (PLEG): container finished" podID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerID="561d3515dc4b5875d0f7ebb4782a807156e78f27b692cd493df2980bca51fc68" exitCode=60
Nov 02 20:30:03.114064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.114036    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerDied","Data":"561d3515dc4b5875d0f7ebb4782a807156e78f27b692cd493df2980bca51fc68"}
Nov 02 20:30:03.114578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.114253    5099 scope.go:122] "RemoveContainer" containerID="561d3515dc4b5875d0f7ebb4782a807156e78f27b692cd493df2980bca51fc68"
Nov 02 20:30:03.116299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.116277    5099 generic.go:356] "Generic (PLEG): container finished" podID="89913e6d-de4b-496e-92ca-fe725e376225" containerID="e7359805e735d57a1a190c3fe88cf8f35f732a37ac382072c786e52f96aa8a14" exitCode=60
Nov 02 20:30:03.116401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.116361    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerDied","Data":"e7359805e735d57a1a190c3fe88cf8f35f732a37ac382072c786e52f96aa8a14"}
Nov 02 20:30:03.116657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.116641    5099 scope.go:122] "RemoveContainer" containerID="e7359805e735d57a1a190c3fe88cf8f35f732a37ac382072c786e52f96aa8a14"
Nov 02 20:30:03.118392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.118375    5099 generic.go:356] "Generic (PLEG): container finished" podID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerID="fb995759a54572ee60fae4ad69f194998992f1108956091ff09bc75db43814eb" exitCode=60
Nov 02 20:30:03.118472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.118402    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerDied","Data":"fb995759a54572ee60fae4ad69f194998992f1108956091ff09bc75db43814eb"}
Nov 02 20:30:03.118735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.118717    5099 scope.go:122] "RemoveContainer" containerID="fb995759a54572ee60fae4ad69f194998992f1108956091ff09bc75db43814eb"
Nov 02 20:30:03.125610 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.125577    5099 generic.go:356] "Generic (PLEG): container finished" podID="48a8397c-80bf-402a-819e-af25eacb69c3" containerID="49d7bdb5f410dc15f0ae66ae9df6173582fc1fc64523de8b94b2a6f66999f7ec" exitCode=60
Nov 02 20:30:03.125733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.125653    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerDied","Data":"49d7bdb5f410dc15f0ae66ae9df6173582fc1fc64523de8b94b2a6f66999f7ec"}
Nov 02 20:30:03.125919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.125901    5099 scope.go:122] "RemoveContainer" containerID="49d7bdb5f410dc15f0ae66ae9df6173582fc1fc64523de8b94b2a6f66999f7ec"
Nov 02 20:30:03.127677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.127660    5099 generic.go:356] "Generic (PLEG): container finished" podID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerID="2f1ac5c5f3ca9023e7647caed8019c33cb1368d372b5445d4c71deb8def14b15" exitCode=60
Nov 02 20:30:03.127766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.127728    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerDied","Data":"2f1ac5c5f3ca9023e7647caed8019c33cb1368d372b5445d4c71deb8def14b15"}
Nov 02 20:30:03.127956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.127940    5099 scope.go:122] "RemoveContainer" containerID="2f1ac5c5f3ca9023e7647caed8019c33cb1368d372b5445d4c71deb8def14b15"
Nov 02 20:30:03.130109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.130090    5099 generic.go:356] "Generic (PLEG): container finished" podID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerID="77a832289b96d37e1f2617fc5b94f92d9d290ee08bf9ef6647f21ac30ab9baea" exitCode=60
Nov 02 20:30:03.130189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.130165    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerDied","Data":"77a832289b96d37e1f2617fc5b94f92d9d290ee08bf9ef6647f21ac30ab9baea"}
Nov 02 20:30:03.130394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.130381    5099 scope.go:122] "RemoveContainer" containerID="77a832289b96d37e1f2617fc5b94f92d9d290ee08bf9ef6647f21ac30ab9baea"
Nov 02 20:30:03.132739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.132720    5099 generic.go:356] "Generic (PLEG): container finished" podID="6df451b8-1cfd-4b23-b289-30257a582317" containerID="cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99" exitCode=0
Nov 02 20:30:03.132966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.132743    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" event={"ID":"6df451b8-1cfd-4b23-b289-30257a582317","Type":"ContainerDied","Data":"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"}
Nov 02 20:30:03.133086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.132985    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv" event={"ID":"6df451b8-1cfd-4b23-b289-30257a582317","Type":"ContainerDied","Data":"05fe958e27a4804a6426c354c640b0e2217c189e246a5dfc234e84856c36ad41"}
Nov 02 20:30:03.133086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.133008    5099 scope.go:122] "RemoveContainer" containerID="cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"
Nov 02 20:30:03.133086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.132765    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"
Nov 02 20:30:03.180860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.180834    5099 scope.go:122] "RemoveContainer" containerID="cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"
Nov 02 20:30:03.184336 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:03.183547    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99\": container with ID starting with cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99 not found: ID does not exist" containerID="cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"
Nov 02 20:30:03.184336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.183588    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99"} err="failed to get container status \"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99\": rpc error: code = NotFound desc = could not find container \"cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99\": container with ID starting with cc0d3f9cbc011daa48aa6234b61f519cdcafb9b456fd67e17e4201a73337fc99 not found: ID does not exist"
Nov 02 20:30:03.336395 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.336361    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:30:03.344845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:03.343075    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-429b45-56bfbc5f79-b72sv"]
Nov 02 20:30:04.146971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.146931    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerStarted","Data":"40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"}
Nov 02 20:30:04.150028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.149992    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerStarted","Data":"f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"}
Nov 02 20:30:04.152664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.152636    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerStarted","Data":"67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"}
Nov 02 20:30:04.155195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.155170    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerStarted","Data":"43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"}
Nov 02 20:30:04.157267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.157237    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerStarted","Data":"17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"}
Nov 02 20:30:04.159446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.159420    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerStarted","Data":"873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"}
Nov 02 20:30:04.161489 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.161465    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerStarted","Data":"6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"}
Nov 02 20:30:04.461950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.461912    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"]
Nov 02 20:30:04.466606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:04.466578    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb060a-9xqv2"]
Nov 02 20:30:05.173699 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.173660    5099 generic.go:356] "Generic (PLEG): container finished" podID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerID="873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59" exitCode=60
Nov 02 20:30:05.173909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.173744    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerDied","Data":"873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"}
Nov 02 20:30:05.173909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.173855    5099 scope.go:122] "RemoveContainer" containerID="2f1ac5c5f3ca9023e7647caed8019c33cb1368d372b5445d4c71deb8def14b15"
Nov 02 20:30:05.174084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.174066    5099 scope.go:122] "RemoveContainer" containerID="873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"
Nov 02 20:30:05.174384 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.174360    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt_odh-ai-gateway-infra(b2ceb221-dbf3-4dac-a0e6-fbc437103f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99"
Nov 02 20:30:05.175759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.175737    5099 generic.go:356] "Generic (PLEG): container finished" podID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerID="6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317" exitCode=60
Nov 02 20:30:05.175874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.175837    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerDied","Data":"6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"}
Nov 02 20:30:05.176101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.176087    5099 scope.go:122] "RemoveContainer" containerID="6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"
Nov 02 20:30:05.176355 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.176338    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n_odh-ai-gateway-infra(a4255fef-8f15-43d4-a8c3-ea8c18519fec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec"
Nov 02 20:30:05.177972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.177949    5099 generic.go:356] "Generic (PLEG): container finished" podID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerID="40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13" exitCode=60
Nov 02 20:30:05.178074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.178053    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerDied","Data":"40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"}
Nov 02 20:30:05.178321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.178303    5099 scope.go:122] "RemoveContainer" containerID="40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"
Nov 02 20:30:05.178647 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.178620    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf_odh-ai-gateway-infra(38efc7ac-860b-4800-9a82-fb84db5f66fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe"
Nov 02 20:30:05.180276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.180252    5099 generic.go:356] "Generic (PLEG): container finished" podID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerID="f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24" exitCode=60
Nov 02 20:30:05.180388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.180326    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerDied","Data":"f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"}
Nov 02 20:30:05.180548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.180530    5099 scope.go:122] "RemoveContainer" containerID="f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"
Nov 02 20:30:05.180818 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.180761    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm_odh-ai-gateway-infra(a2ffc430-c7ba-4778-ad05-5b67f8af3219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219"
Nov 02 20:30:05.182711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.182689    5099 generic.go:356] "Generic (PLEG): container finished" podID="89913e6d-de4b-496e-92ca-fe725e376225" containerID="67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3" exitCode=60
Nov 02 20:30:05.182856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.182766    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerDied","Data":"67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"}
Nov 02 20:30:05.183056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.183038    5099 scope.go:122] "RemoveContainer" containerID="67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"
Nov 02 20:30:05.183320 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.183294    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-zvzf9_odh-ai-gateway-infra(89913e6d-de4b-496e-92ca-fe725e376225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" podUID="89913e6d-de4b-496e-92ca-fe725e376225"
Nov 02 20:30:05.184949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.184919    5099 generic.go:356] "Generic (PLEG): container finished" podID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerID="43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a" exitCode=60
Nov 02 20:30:05.185065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.184948    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerDied","Data":"43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"}
Nov 02 20:30:05.185217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.185199    5099 scope.go:122] "RemoveContainer" containerID="43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"
Nov 02 20:30:05.185517 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.185495    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb_odh-ai-gateway-infra(3476cb95-0386-45f9-b490-e0c5f7fbabb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7"
Nov 02 20:30:05.187362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.187341    5099 generic.go:356] "Generic (PLEG): container finished" podID="48a8397c-80bf-402a-819e-af25eacb69c3" containerID="17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5" exitCode=60
Nov 02 20:30:05.187453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.187378    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerDied","Data":"17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"}
Nov 02 20:30:05.187667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.187648    5099 scope.go:122] "RemoveContainer" containerID="17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"
Nov 02 20:30:05.187991 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:05.187957    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj_odh-ai-gateway-infra(48a8397c-80bf-402a-819e-af25eacb69c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" podUID="48a8397c-80bf-402a-819e-af25eacb69c3"
Nov 02 20:30:05.190279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.190257    5099 scope.go:122] "RemoveContainer" containerID="77a832289b96d37e1f2617fc5b94f92d9d290ee08bf9ef6647f21ac30ab9baea"
Nov 02 20:30:05.213087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.213063    5099 scope.go:122] "RemoveContainer" containerID="09b2035195c4ae8315d3cd67406e0a6a1ef118683354aa9b84fba3aeb551aeee"
Nov 02 20:30:05.230390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.230366    5099 scope.go:122] "RemoveContainer" containerID="561d3515dc4b5875d0f7ebb4782a807156e78f27b692cd493df2980bca51fc68"
Nov 02 20:30:05.245654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.245523    5099 scope.go:122] "RemoveContainer" containerID="e7359805e735d57a1a190c3fe88cf8f35f732a37ac382072c786e52f96aa8a14"
Nov 02 20:30:05.260338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.260308    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df451b8-1cfd-4b23-b289-30257a582317" path="/var/lib/kubelet/pods/6df451b8-1cfd-4b23-b289-30257a582317/volumes"
Nov 02 20:30:05.260934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.260911    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82e979a-2479-420a-abfe-8e7f80b746c1" path="/var/lib/kubelet/pods/b82e979a-2479-420a-abfe-8e7f80b746c1/volumes"
Nov 02 20:30:05.263050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.263030    5099 scope.go:122] "RemoveContainer" containerID="fb995759a54572ee60fae4ad69f194998992f1108956091ff09bc75db43814eb"
Nov 02 20:30:05.278122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:05.278100    5099 scope.go:122] "RemoveContainer" containerID="49d7bdb5f410dc15f0ae66ae9df6173582fc1fc64523de8b94b2a6f66999f7ec"
Nov 02 20:30:06.198688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.198643    5099 scope.go:122] "RemoveContainer" containerID="f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"
Nov 02 20:30:06.199124 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.199005    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm_odh-ai-gateway-infra(a2ffc430-c7ba-4778-ad05-5b67f8af3219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219"
Nov 02 20:30:06.201255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.201233    5099 scope.go:122] "RemoveContainer" containerID="67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"
Nov 02 20:30:06.201571 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.201538    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-zvzf9_odh-ai-gateway-infra(89913e6d-de4b-496e-92ca-fe725e376225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" podUID="89913e6d-de4b-496e-92ca-fe725e376225"
Nov 02 20:30:06.202972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.202951    5099 scope.go:122] "RemoveContainer" containerID="43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"
Nov 02 20:30:06.203280 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.203244    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb_odh-ai-gateway-infra(3476cb95-0386-45f9-b490-e0c5f7fbabb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7"
Nov 02 20:30:06.204586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.204566    5099 scope.go:122] "RemoveContainer" containerID="17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"
Nov 02 20:30:06.204836 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.204811    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj_odh-ai-gateway-infra(48a8397c-80bf-402a-819e-af25eacb69c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" podUID="48a8397c-80bf-402a-819e-af25eacb69c3"
Nov 02 20:30:06.206456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.206435    5099 scope.go:122] "RemoveContainer" containerID="873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"
Nov 02 20:30:06.206886 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.206701    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt_odh-ai-gateway-infra(b2ceb221-dbf3-4dac-a0e6-fbc437103f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99"
Nov 02 20:30:06.208829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.208809    5099 scope.go:122] "RemoveContainer" containerID="6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"
Nov 02 20:30:06.209066 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.209045    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n_odh-ai-gateway-infra(a4255fef-8f15-43d4-a8c3-ea8c18519fec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec"
Nov 02 20:30:06.210611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:06.210596    5099 scope.go:122] "RemoveContainer" containerID="40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"
Nov 02 20:30:06.210812 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:06.210776    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf_odh-ai-gateway-infra(38efc7ac-860b-4800-9a82-fb84db5f66fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe"
Nov 02 20:30:15.250847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:15.250818    5099 scope.go:122] "RemoveContainer" containerID="43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"
Nov 02 20:30:16.096270 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:16.096231    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:30:16.250637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.250610    5099 scope.go:122] "RemoveContainer" containerID="67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"
Nov 02 20:30:16.250855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.250833    5099 scope.go:122] "RemoveContainer" containerID="873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"
Nov 02 20:30:16.308201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.308171    5099 generic.go:356] "Generic (PLEG): container finished" podID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerID="28abf553f669f7285105d873797302ecf29645352d75b5d62fe373ab561a1f85" exitCode=60
Nov 02 20:30:16.308615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.308245    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerDied","Data":"28abf553f669f7285105d873797302ecf29645352d75b5d62fe373ab561a1f85"}
Nov 02 20:30:16.308615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.308302    5099 scope.go:122] "RemoveContainer" containerID="43298aea718f313e3549352cf13aa7a9eda8d6a064f7e2e74d1ddf38a350c38a"
Nov 02 20:30:16.308795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:16.308768    5099 scope.go:122] "RemoveContainer" containerID="28abf553f669f7285105d873797302ecf29645352d75b5d62fe373ab561a1f85"
Nov 02 20:30:16.309235 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:16.309151    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb_odh-ai-gateway-infra(3476cb95-0386-45f9-b490-e0c5f7fbabb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7"
Nov 02 20:30:17.250569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.250523    5099 scope.go:122] "RemoveContainer" containerID="f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"
Nov 02 20:30:17.320012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.319978    5099 generic.go:356] "Generic (PLEG): container finished" podID="89913e6d-de4b-496e-92ca-fe725e376225" containerID="cd26566eec2a45f4dda9806ba58dc8944138ef53c9340112be0ba5285a2a8b40" exitCode=60
Nov 02 20:30:17.320451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.320018    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerDied","Data":"cd26566eec2a45f4dda9806ba58dc8944138ef53c9340112be0ba5285a2a8b40"}
Nov 02 20:30:17.320451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.320079    5099 scope.go:122] "RemoveContainer" containerID="67716fe35e0ebc2123ff6c7f3bae410d5fad5bc03c91bfaca4746800f65247d3"
Nov 02 20:30:17.320730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.320542    5099 scope.go:122] "RemoveContainer" containerID="cd26566eec2a45f4dda9806ba58dc8944138ef53c9340112be0ba5285a2a8b40"
Nov 02 20:30:17.321016 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:17.320986    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-zvzf9_odh-ai-gateway-infra(89913e6d-de4b-496e-92ca-fe725e376225)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" podUID="89913e6d-de4b-496e-92ca-fe725e376225"
Nov 02 20:30:17.324020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.324002    5099 generic.go:356] "Generic (PLEG): container finished" podID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerID="c45797308634c17df341f4af14f15356f457379cb1e4995ce8d79e9d4c5ec4c4" exitCode=60
Nov 02 20:30:17.324143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.324040    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerDied","Data":"c45797308634c17df341f4af14f15356f457379cb1e4995ce8d79e9d4c5ec4c4"}
Nov 02 20:30:17.324409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.324395    5099 scope.go:122] "RemoveContainer" containerID="c45797308634c17df341f4af14f15356f457379cb1e4995ce8d79e9d4c5ec4c4"
Nov 02 20:30:17.324724 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:17.324706    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt_odh-ai-gateway-infra(b2ceb221-dbf3-4dac-a0e6-fbc437103f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99"
Nov 02 20:30:17.335258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.335236    5099 scope.go:122] "RemoveContainer" containerID="873b0506ff3bf7a920b72bf80ca328fa3d0c215cb4428e58cb021f30216aaa59"
Nov 02 20:30:17.427970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.427865    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:17.641173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.641144    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:17.679262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.679186    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\" (UniqueName: \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\") pod \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") "
Nov 02 20:30:17.679432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.679367    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\") pod \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\" (UID: \"3476cb95-0386-45f9-b490-e0c5f7fbabb7\") "
Nov 02 20:30:17.679777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.679747    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca" pod "3476cb95-0386-45f9-b490-e0c5f7fbabb7" (UID: "3476cb95-0386-45f9-b490-e0c5f7fbabb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:17.680045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.680023    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3476cb95-0386-45f9-b490-e0c5f7fbabb7-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:17.681427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.681399    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk" pod "3476cb95-0386-45f9-b490-e0c5f7fbabb7" (UID: "3476cb95-0386-45f9-b490-e0c5f7fbabb7"). InnerVolumeSpecName "kube-api-access-7xpsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:17.781400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:17.781364    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpsk\" (UniqueName: \"kubernetes.io/projected/3476cb95-0386-45f9-b490-e0c5f7fbabb7-kube-api-access-7xpsk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:18.251654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.250821    5099 scope.go:122] "RemoveContainer" containerID="17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"
Nov 02 20:30:18.251997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.251906    5099 scope.go:122] "RemoveContainer" containerID="40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"
Nov 02 20:30:18.335737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.335664    5099 generic.go:356] "Generic (PLEG): container finished" podID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerID="843ab46df99f4c978db54a23b115b63cea8cac204758557f53de02e6db1547af" exitCode=60
Nov 02 20:30:18.336212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.335738    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerDied","Data":"843ab46df99f4c978db54a23b115b63cea8cac204758557f53de02e6db1547af"}
Nov 02 20:30:18.336212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.335834    5099 scope.go:122] "RemoveContainer" containerID="f30ef1aa21f8edef67c9005f9e7048d0713ba75e9df0cba57500efd4ce7e2e24"
Nov 02 20:30:18.336319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.336269    5099 scope.go:122] "RemoveContainer" containerID="843ab46df99f4c978db54a23b115b63cea8cac204758557f53de02e6db1547af"
Nov 02 20:30:18.337084 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:18.337056    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm_odh-ai-gateway-infra(a2ffc430-c7ba-4778-ad05-5b67f8af3219)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219"
Nov 02 20:30:18.339902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.339877    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb" event={"ID":"3476cb95-0386-45f9-b490-e0c5f7fbabb7","Type":"ContainerDied","Data":"0068bbd8ec2a3e0395ed508bd7f4227f7351c7b06b59a74c31ebf14f636e5afe"}
Nov 02 20:30:18.340033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.339906    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"
Nov 02 20:30:18.353219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.353199    5099 scope.go:122] "RemoveContainer" containerID="28abf553f669f7285105d873797302ecf29645352d75b5d62fe373ab561a1f85"
Nov 02 20:30:18.556383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.556351    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:18.582250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.582224    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:18.592307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.592243    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:18.598738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:18.598711    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ae8e6e-29894190-mdnbb"]
Nov 02 20:30:19.031511 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.031472    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:30:19.033716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033685    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.033716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033714    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033748    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033757    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033770    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6df451b8-1cfd-4b23-b289-30257a582317" containerName="payload-processing"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033792    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df451b8-1cfd-4b23-b289-30257a582317" containerName="payload-processing"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033815    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.033971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.033822    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.034302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.034079    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.034302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.034096    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.034302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.034106    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df451b8-1cfd-4b23-b289-30257a582317" containerName="payload-processing"
Nov 02 20:30:19.034302 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.034118    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" containerName="cleanup"
Nov 02 20:30:19.073166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.073135    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.098275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.098076    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:30:19.115889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.115859    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:30:19.126550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.126508    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwmb\" (UniqueName: \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.126888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.126864    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.127036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.127017    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:30:19.127116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.127105    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.127578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.127477    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-65f28f\""
Nov 02 20:30:19.127578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.127532    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-65f28f-dockercfg-l5qnv\""
Nov 02 20:30:19.230592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.230543    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwmb\" (UniqueName: \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.230773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.230641    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.230773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.230740    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.230963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.230882    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkb6\" (UniqueName: \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.232300 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.232267    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.254706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.254669    5099 scope.go:122] "RemoveContainer" containerID="6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"
Nov 02 20:30:19.254977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.254954    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:19.295995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.295964    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwmb\" (UniqueName: \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\") pod \"payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.302022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.301981    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3476cb95-0386-45f9-b490-e0c5f7fbabb7" path="/var/lib/kubelet/pods/3476cb95-0386-45f9-b490-e0c5f7fbabb7/volumes"
Nov 02 20:30:19.332110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.332072    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.332290 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.332270    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkb6\" (UniqueName: \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.333684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.333660    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.364638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.364528    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerStarted","Data":"36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1"}
Nov 02 20:30:19.365091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.365037    5099 scope.go:122] "RemoveContainer" containerID="36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1"
Nov 02 20:30:19.365463 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:19.365434    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj_odh-ai-gateway-infra(48a8397c-80bf-402a-819e-af25eacb69c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" podUID="48a8397c-80bf-402a-819e-af25eacb69c3"
Nov 02 20:30:19.366371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.366319    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt" event={"ID":"b2ceb221-dbf3-4dac-a0e6-fbc437103f99","Type":"ContainerDied","Data":"8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1"}
Nov 02 20:30:19.366521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.366374    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8380dffb843549b1be33206d70e724b8d46e1f6b5cae7c27a5b0fbb1f4ba56c1"
Nov 02 20:30:19.369046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.369021    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerStarted","Data":"7544c8b3bc30fe818c774a0f9ace655691b4b7eb795b2cb5d99342dfdc438ec6"}
Nov 02 20:30:19.373347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.373307    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9" event={"ID":"89913e6d-de4b-496e-92ca-fe725e376225","Type":"ContainerDied","Data":"3d230d743898585ff6f3f13a528f53de70eae42088c29d50d8a80409fd56f01a"}
Nov 02 20:30:19.373455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.373311    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"
Nov 02 20:30:19.373567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.373375    5099 scope.go:122] "RemoveContainer" containerID="cd26566eec2a45f4dda9806ba58dc8944138ef53c9340112be0ba5285a2a8b40"
Nov 02 20:30:19.389007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.388986    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkb6\" (UniqueName: \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\") pod \"payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.410627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.410600    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:19.431090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.431064    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:19.435749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.433903    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\") pod \"89913e6d-de4b-496e-92ca-fe725e376225\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") "
Nov 02 20:30:19.435749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.434117    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\" (UniqueName: \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\") pod \"89913e6d-de4b-496e-92ca-fe725e376225\" (UID: \"89913e6d-de4b-496e-92ca-fe725e376225\") "
Nov 02 20:30:19.435749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.434322    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca" pod "89913e6d-de4b-496e-92ca-fe725e376225" (UID: "89913e6d-de4b-496e-92ca-fe725e376225"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:19.435749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.435456    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89913e6d-de4b-496e-92ca-fe725e376225-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.439389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.439368    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j" pod "89913e6d-de4b-496e-92ca-fe725e376225" (UID: "89913e6d-de4b-496e-92ca-fe725e376225"). InnerVolumeSpecName "kube-api-access-rnc2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:19.460270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.459926    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:19.471528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.471293    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:19.536856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.536796    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\") pod \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") "
Nov 02 20:30:19.537056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.536955    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\" (UniqueName: \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\") pod \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\" (UID: \"b2ceb221-dbf3-4dac-a0e6-fbc437103f99\") "
Nov 02 20:30:19.537682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.537656    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnc2j\" (UniqueName: \"kubernetes.io/projected/89913e6d-de4b-496e-92ca-fe725e376225-kube-api-access-rnc2j\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.538499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.538457    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca" pod "b2ceb221-dbf3-4dac-a0e6-fbc437103f99" (UID: "b2ceb221-dbf3-4dac-a0e6-fbc437103f99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:19.546432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.546398    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn" pod "b2ceb221-dbf3-4dac-a0e6-fbc437103f99" (UID: "b2ceb221-dbf3-4dac-a0e6-fbc437103f99"). InnerVolumeSpecName "kube-api-access-dmcbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:19.641058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.640451    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.641058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.640488    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmcbn\" (UniqueName: \"kubernetes.io/projected/b2ceb221-dbf3-4dac-a0e6-fbc437103f99-kube-api-access-dmcbn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.888412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.888348    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:19.945442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.945413    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\") pod \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") "
Nov 02 20:30:19.945643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.945457    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\" (UniqueName: \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\") pod \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\" (UID: \"a2ffc430-c7ba-4778-ad05-5b67f8af3219\") "
Nov 02 20:30:19.945884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.945855    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca" pod "a2ffc430-c7ba-4778-ad05-5b67f8af3219" (UID: "a2ffc430-c7ba-4778-ad05-5b67f8af3219"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:19.946334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.946312    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffc430-c7ba-4778-ad05-5b67f8af3219-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:19.947738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.947712    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk" pod "a2ffc430-c7ba-4778-ad05-5b67f8af3219" (UID: "a2ffc430-c7ba-4778-ad05-5b67f8af3219"). InnerVolumeSpecName "kube-api-access-pxwmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:19.996228 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:30:19.996201    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f3557f_7978_46c2_aa99_c6ecb3384492.slice/crio-347686dc9a1406250dc129f9c253d68817c48be48919e319085748840927234c WatchSource:0}: Error finding container 347686dc9a1406250dc129f9c253d68817c48be48919e319085748840927234c: Status 404 returned error can't find the container with id 347686dc9a1406250dc129f9c253d68817c48be48919e319085748840927234c
Nov 02 20:30:19.996773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:19.996748    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:30:20.047329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.047288    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:30:20.048286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.048258    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwmk\" (UniqueName: \"kubernetes.io/projected/a2ffc430-c7ba-4778-ad05-5b67f8af3219-kube-api-access-pxwmk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:20.114747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.114717    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:20.119807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.119767    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-zvzf9"]
Nov 02 20:30:20.386754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.386725    5099 generic.go:356] "Generic (PLEG): container finished" podID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerID="b5ddc99083b06fb85d3072fea1e7139cc26d863765e38fd9dbfd21c91f169afe" exitCode=60
Nov 02 20:30:20.387227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.386827    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerDied","Data":"b5ddc99083b06fb85d3072fea1e7139cc26d863765e38fd9dbfd21c91f169afe"}
Nov 02 20:30:20.387227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.386892    5099 scope.go:122] "RemoveContainer" containerID="6e2bf56c12a1f7192e5c4d0c49c6b20505c3287413ec1ceb033fecc42aa4f317"
Nov 02 20:30:20.387393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.387375    5099 scope.go:122] "RemoveContainer" containerID="b5ddc99083b06fb85d3072fea1e7139cc26d863765e38fd9dbfd21c91f169afe"
Nov 02 20:30:20.388211 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:20.387825    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n_odh-ai-gateway-infra(a4255fef-8f15-43d4-a8c3-ea8c18519fec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec"
Nov 02 20:30:20.389014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.388992    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" event={"ID":"9c448487-d84f-44fd-b7e4-a2ec1635bc62","Type":"ContainerStarted","Data":"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"}
Nov 02 20:30:20.389114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.389037    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" event={"ID":"9c448487-d84f-44fd-b7e4-a2ec1635bc62","Type":"ContainerStarted","Data":"55d43c37d6155cfd5f97bf89824bb76b04f37d2e6ce8bf89e379094ae6fda9b3"}
Nov 02 20:30:20.389114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.389087    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:20.391744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.391725    5099 generic.go:356] "Generic (PLEG): container finished" podID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerID="7544c8b3bc30fe818c774a0f9ace655691b4b7eb795b2cb5d99342dfdc438ec6" exitCode=60
Nov 02 20:30:20.391844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.391823    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerDied","Data":"7544c8b3bc30fe818c774a0f9ace655691b4b7eb795b2cb5d99342dfdc438ec6"}
Nov 02 20:30:20.392223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.392201    5099 scope.go:122] "RemoveContainer" containerID="7544c8b3bc30fe818c774a0f9ace655691b4b7eb795b2cb5d99342dfdc438ec6"
Nov 02 20:30:20.392534 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:20.392510    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf_odh-ai-gateway-infra(38efc7ac-860b-4800-9a82-fb84db5f66fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe"
Nov 02 20:30:20.393226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.393208    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"
Nov 02 20:30:20.393327 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.393248    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm" event={"ID":"a2ffc430-c7ba-4778-ad05-5b67f8af3219","Type":"ContainerDied","Data":"147137ea0fbe60ee6430bba9aed3225d01b5023c857d6040ed2005b34b708bec"}
Nov 02 20:30:20.396373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.396327    5099 generic.go:356] "Generic (PLEG): container finished" podID="48a8397c-80bf-402a-819e-af25eacb69c3" containerID="36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1" exitCode=60
Nov 02 20:30:20.396452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.396385    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerDied","Data":"36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1"}
Nov 02 20:30:20.396667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.396650    5099 scope.go:122] "RemoveContainer" containerID="36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1"
Nov 02 20:30:20.396986 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:20.396965    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj_odh-ai-gateway-infra(48a8397c-80bf-402a-819e-af25eacb69c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" podUID="48a8397c-80bf-402a-819e-af25eacb69c3"
Nov 02 20:30:20.398035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.398011    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" event={"ID":"42f3557f-7978-46c2-aa99-c6ecb3384492","Type":"ContainerStarted","Data":"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"}
Nov 02 20:30:20.398122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.398049    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" event={"ID":"42f3557f-7978-46c2-aa99-c6ecb3384492","Type":"ContainerStarted","Data":"347686dc9a1406250dc129f9c253d68817c48be48919e319085748840927234c"}
Nov 02 20:30:20.398122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.398020    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"
Nov 02 20:30:20.398233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.398084    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:20.411190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.411172    5099 scope.go:122] "RemoveContainer" containerID="40b0cfab57cecb41babcbfefc03500134a2a9f97fe5c1f37fbd3ead00b6ebc13"
Nov 02 20:30:20.425729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.425709    5099 scope.go:122] "RemoveContainer" containerID="843ab46df99f4c978db54a23b115b63cea8cac204758557f53de02e6db1547af"
Nov 02 20:30:20.435323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.435299    5099 scope.go:122] "RemoveContainer" containerID="17b6134c9128106324b53f3d53df056a515258bcd4a04e4e1150612a6ea6dbc5"
Nov 02 20:30:20.526373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.526334    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:20.690304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.690220    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:20.714214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.714150    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" podStartSLOduration=2.714136325 podStartE2EDuration="2.714136325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:20.709021607 +0000 UTC m=+2309.583292494" watchObservedRunningTime="2026-11-02 20:30:20.714136325 +0000 UTC m=+2309.588407213"
Nov 02 20:30:20.819632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.818388    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:20.879806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:20.870472    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-982b8a-29894190-xwsnt"]
Nov 02 20:30:21.074407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.074286    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" podStartSLOduration=3.074269272 podStartE2EDuration="3.074269272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:21.07246825 +0000 UTC m=+2309.946739139" watchObservedRunningTime="2026-11-02 20:30:21.074269272 +0000 UTC m=+2309.948540163"
Nov 02 20:30:21.187517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.182907    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:21.208816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.203919    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-be1348-29894190-mtdxm"]
Nov 02 20:30:21.303817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.303765    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89913e6d-de4b-496e-92ca-fe725e376225" path="/var/lib/kubelet/pods/89913e6d-de4b-496e-92ca-fe725e376225/volumes"
Nov 02 20:30:21.304429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.304404    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" path="/var/lib/kubelet/pods/a2ffc430-c7ba-4778-ad05-5b67f8af3219/volumes"
Nov 02 20:30:21.304985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.304967    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" path="/var/lib/kubelet/pods/b2ceb221-dbf3-4dac-a0e6-fbc437103f99/volumes"
Nov 02 20:30:21.513695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.513242    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:21.628814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.626989    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:21.706670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.706638    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:21.708905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.708878    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\" (UniqueName: \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\") pod \"38efc7ac-860b-4800-9a82-fb84db5f66fe\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") "
Nov 02 20:30:21.709059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.708924    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\") pod \"38efc7ac-860b-4800-9a82-fb84db5f66fe\" (UID: \"38efc7ac-860b-4800-9a82-fb84db5f66fe\") "
Nov 02 20:30:21.709139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.709122    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\") pod \"48a8397c-80bf-402a-819e-af25eacb69c3\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") "
Nov 02 20:30:21.709416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.709383    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca" pod "38efc7ac-860b-4800-9a82-fb84db5f66fe" (UID: "38efc7ac-860b-4800-9a82-fb84db5f66fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:21.709533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.709456    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca" pod "48a8397c-80bf-402a-819e-af25eacb69c3" (UID: "48a8397c-80bf-402a-819e-af25eacb69c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:21.709818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.709798    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a8397c-80bf-402a-819e-af25eacb69c3-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:21.709896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.709825    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38efc7ac-860b-4800-9a82-fb84db5f66fe-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:21.711727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.711700    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p" pod "38efc7ac-860b-4800-9a82-fb84db5f66fe" (UID: "38efc7ac-860b-4800-9a82-fb84db5f66fe"). InnerVolumeSpecName "kube-api-access-5b97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:21.784893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.784862    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:21.811370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.811337    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\") pod \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") "
Nov 02 20:30:21.811553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.811522    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\" (UniqueName: \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\") pod \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\" (UID: \"a4255fef-8f15-43d4-a8c3-ea8c18519fec\") "
Nov 02 20:30:21.811617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.811574    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\" (UniqueName: \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\") pod \"48a8397c-80bf-402a-819e-af25eacb69c3\" (UID: \"48a8397c-80bf-402a-819e-af25eacb69c3\") "
Nov 02 20:30:21.811712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.811687    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca" pod "a4255fef-8f15-43d4-a8c3-ea8c18519fec" (UID: "a4255fef-8f15-43d4-a8c3-ea8c18519fec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:30:21.812152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.812128    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b97p\" (UniqueName: \"kubernetes.io/projected/38efc7ac-860b-4800-9a82-fb84db5f66fe-kube-api-access-5b97p\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:21.812152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.812150    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4255fef-8f15-43d4-a8c3-ea8c18519fec-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:21.813752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.813727    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv" pod "a4255fef-8f15-43d4-a8c3-ea8c18519fec" (UID: "a4255fef-8f15-43d4-a8c3-ea8c18519fec"). InnerVolumeSpecName "kube-api-access-66nnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:21.814094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.814073    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9" pod "48a8397c-80bf-402a-819e-af25eacb69c3" (UID: "48a8397c-80bf-402a-819e-af25eacb69c3"). InnerVolumeSpecName "kube-api-access-2wrd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:21.913298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.913264    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nnv\" (UniqueName: \"kubernetes.io/projected/a4255fef-8f15-43d4-a8c3-ea8c18519fec-kube-api-access-66nnv\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:21.913298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:21.913289    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrd9\" (UniqueName: \"kubernetes.io/projected/48a8397c-80bf-402a-819e-af25eacb69c3-kube-api-access-2wrd9\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:22.423832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.423798    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"
Nov 02 20:30:22.423832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.423817    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf" event={"ID":"38efc7ac-860b-4800-9a82-fb84db5f66fe","Type":"ContainerDied","Data":"888e2742f22d2e6ca2f08db8badcffb06e930b03d3088cc9a51b64aef977f2df"}
Nov 02 20:30:22.424122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.423888    5099 scope.go:122] "RemoveContainer" containerID="7544c8b3bc30fe818c774a0f9ace655691b4b7eb795b2cb5d99342dfdc438ec6"
Nov 02 20:30:22.425584 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.425561    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"
Nov 02 20:30:22.425731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.425583    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj" event={"ID":"48a8397c-80bf-402a-819e-af25eacb69c3","Type":"ContainerDied","Data":"63415f7b41e55678feb2d6b4d7e26f9b78e0803c5fa8539ce3dac286bd69010d"}
Nov 02 20:30:22.427035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.427015    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n" event={"ID":"a4255fef-8f15-43d4-a8c3-ea8c18519fec","Type":"ContainerDied","Data":"906d191ed8f0fd41b8fbe69266a4dde7da91b62e618083a3e7a2fc7b84400dd0"}
Nov 02 20:30:22.427145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.427041    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"
Nov 02 20:30:22.439734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.439709    5099 scope.go:122] "RemoveContainer" containerID="36772a3d467f1a326c2fe4cb73795f62d82a0793b8ee8249f7a71b76d9b9acb1"
Nov 02 20:30:22.454090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.454070    5099 scope.go:122] "RemoveContainer" containerID="b5ddc99083b06fb85d3072fea1e7139cc26d863765e38fd9dbfd21c91f169afe"
Nov 02 20:30:22.584842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.584798    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:22.589885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.589860    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894190-795rj"]
Nov 02 20:30:22.689761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.689678    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:22.701673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.701643    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-429b45-29894190-twh5n"]
Nov 02 20:30:22.804028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.803994    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:22.809902 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:22.809875    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5efe19-29894190-2zdvf"]
Nov 02 20:30:23.262407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:23.262371    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" path="/var/lib/kubelet/pods/38efc7ac-860b-4800-9a82-fb84db5f66fe/volumes"
Nov 02 20:30:23.262963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:23.262942    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" path="/var/lib/kubelet/pods/48a8397c-80bf-402a-819e-af25eacb69c3/volumes"
Nov 02 20:30:23.263447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:23.263432    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" path="/var/lib/kubelet/pods/a4255fef-8f15-43d4-a8c3-ea8c18519fec/volumes"
Nov 02 20:30:31.492351 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.492320    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:30:31.497849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.494128    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:30:31.715933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.715899    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:30:31.717775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717740    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.717775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717771    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.717775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717810    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.717775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717819    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.717775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717832    5099 cpu_manager.go:414] "containerMap: removing container" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717840    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717853    5099 cpu_manager.go:414] "containerMap: removing container" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717861    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717871    5099 cpu_manager.go:414] "containerMap: removing container" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717879    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717894    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717901    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717913    5099 cpu_manager.go:414] "containerMap: removing container" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717919    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717931    5099 cpu_manager.go:414] "containerMap: removing container" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717938    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717956    5099 cpu_manager.go:414] "containerMap: removing container" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717963    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717982    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.717989    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718013    5099 cpu_manager.go:414] "containerMap: removing container" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718020    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718032    5099 cpu_manager.go:414] "containerMap: removing container" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718039    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718066    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718074    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718085    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.718195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718091    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718299    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718310    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718323    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718332    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718342    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718350    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718360    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718371    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718379    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718389    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718395    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718407    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8397c-80bf-402a-819e-af25eacb69c3" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718415    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.719391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.718423    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.735346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.735302    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:31.738820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.738776    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:30:31.805587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.804917    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb78ba\""
Nov 02 20:30:31.805587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.805061    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bb78ba-dockercfg-f666j\""
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.816166    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817542    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817562    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817581    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817589    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4255fef-8f15-43d4-a8c3-ea8c18519fec" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817615    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817623    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817639    5099 cpu_manager.go:414] "containerMap: removing container" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817646    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817891    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="38efc7ac-860b-4800-9a82-fb84db5f66fe" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817905    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ffc430-c7ba-4778-ad05-5b67f8af3219" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817920    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="89913e6d-de4b-496e-92ca-fe725e376225" containerName="cleanup"
Nov 02 20:30:31.818682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.817934    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ceb221-dbf3-4dac-a0e6-fbc437103f99" containerName="cleanup"
Nov 02 20:30:31.825039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.825017    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:31.857439 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:31.857399    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:30:32.038884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.029926    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.038884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.030026    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgs8f\" (UniqueName: \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.038884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.030107    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwzt\" (UniqueName: \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.038884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.030232    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.131501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.131468    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgs8f\" (UniqueName: \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.131677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.131560    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwzt\" (UniqueName: \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.131742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.131684    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.131860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.131841    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.132512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.132485    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.132657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.132595    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.185873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.185842    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwzt\" (UniqueName: \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\") pod \"payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.186059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.186035    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgs8f\" (UniqueName: \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\") pod \"payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.416382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.416350    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:32.442408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.442373    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:32.854047 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.853965    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:30:32.906487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:32.906411    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:30:33.624601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.624552    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" event={"ID":"94eb361c-ab36-4bda-be4a-01a99f73a620","Type":"ContainerStarted","Data":"780d4c275bdbca8cd2fe0deefe1bbc2f8e54dfbef44f2246f610f54bc522338c"}
Nov 02 20:30:33.624830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.624627    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" event={"ID":"94eb361c-ab36-4bda-be4a-01a99f73a620","Type":"ContainerStarted","Data":"f78b9f645b69daa5737877d83569dfabb09449f1b46d3548718340e03ed2a9e0"}
Nov 02 20:30:33.624954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.624907    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:33.629836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.627866    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" event={"ID":"97063b16-3959-4ea1-866c-219bc6460898","Type":"ContainerStarted","Data":"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"}
Nov 02 20:30:33.629836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.627921    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" event={"ID":"97063b16-3959-4ea1-866c-219bc6460898","Type":"ContainerStarted","Data":"f140a303b234f57d86cb30b129c733af69480f49dffb22a7a1b55bad5c708c21"}
Nov 02 20:30:33.629836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.628534    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:33.736822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.736714    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" podStartSLOduration=2.736694785 podStartE2EDuration="2.736694785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:33.732778058 +0000 UTC m=+2322.607048948" watchObservedRunningTime="2026-11-02 20:30:33.736694785 +0000 UTC m=+2322.610965675"
Nov 02 20:30:33.839840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:33.839762    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" podStartSLOduration=2.839744178 podStartE2EDuration="2.839744178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:33.839292818 +0000 UTC m=+2322.713563707" watchObservedRunningTime="2026-11-02 20:30:33.839744178 +0000 UTC m=+2322.714015066"
Nov 02 20:30:39.998586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:39.998552    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:30:44.641537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:44.641500    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:30:45.648803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:45.648761    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:30:46.099941 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:30:46.099861    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:30:55.021497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.021456    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main" containerID="cri-o://b3eb25fcc9d18ab1c2bebae4a56d2d0623c7f9fd70fab0f9c18c677a2f98dcc7" gracePeriod=45
Nov 02 20:30:55.313811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.313757    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:30:55.342346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.342313    5099 scope.go:122] "RemoveContainer" containerID="b3eb25fcc9d18ab1c2bebae4a56d2d0623c7f9fd70fab0f9c18c677a2f98dcc7"
Nov 02 20:30:55.431130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431100    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\" (UniqueName: \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431228    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431297 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431279    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431309    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431336    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431439    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\") pod \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\" (UID: \"b6319c39-77fd-4bdb-aaa1-a70cb4551c87\") "
Nov 02 20:30:55.431553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431518    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:55.431882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.431753    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:55.432744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.432107    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:55.432744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.432473    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.432744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.432501    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.432744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.432514    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.434151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.434119    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "kube-api-access-dq9fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:30:55.434266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.434205    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:30:55.435979 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.435942    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm" pod "b6319c39-77fd-4bdb-aaa1-a70cb4551c87" (UID: "b6319c39-77fd-4bdb-aaa1-a70cb4551c87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:30:55.536260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.535079    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9fw\" (UniqueName: \"kubernetes.io/projected/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-kube-api-access-dq9fw\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.536260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.535118    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.536260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.535135    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6319c39-77fd-4bdb-aaa1-a70cb4551c87-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:30:55.833982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.833897    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"
Nov 02 20:30:55.833982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.833914    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" event={"ID":"b6319c39-77fd-4bdb-aaa1-a70cb4551c87","Type":"ContainerDied","Data":"b3eb25fcc9d18ab1c2bebae4a56d2d0623c7f9fd70fab0f9c18c677a2f98dcc7"}
Nov 02 20:30:55.833982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:55.833956    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" event={"ID":"b6319c39-77fd-4bdb-aaa1-a70cb4551c87","Type":"ContainerDied","Data":"d4b9b2882f4a7d38ad9e54039dc0b2be382018d845dc4b0be9f62c871170125c"}
Nov 02 20:30:56.049134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:56.049079    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:30:56.053138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:56.053110    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf"]
Nov 02 20:30:56.266888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:56.266838    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-602be312/e2e-lifecycle-model-602be312-kserve-698fc64ffc-87zhf" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:30:57.260534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:30:57.260455    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" path="/var/lib/kubelet/pods/b6319c39-77fd-4bdb-aaa1-a70cb4551c87/volumes"
Nov 02 20:31:13.224945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:13.224903    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"]
Nov 02 20:31:13.318375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:13.318337    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c76017-h9wsn"]
Nov 02 20:31:15.261992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:15.261957    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8257ab67-825f-4457-b8f3-06d5069ea61d" path="/var/lib/kubelet/pods/8257ab67-825f-4457-b8f3-06d5069ea61d/volumes"
Nov 02 20:31:16.096320 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:16.096287    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:31:19.859335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:19.859294    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:31:19.859956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:19.859548    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy" containerID="cri-o://2a99b5cffcb65874f60c17b2b88b386bb1ee72bc24d076fe0a8c8b382a9f9746" gracePeriod=30
Nov 02 20:31:21.434701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:21.434665    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:31:21.435200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:21.434960    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" containerID="cri-o://dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd" gracePeriod=30
Nov 02 20:31:22.879214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.879178    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:31:22.879600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.879398    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" containerID="cri-o://23f8a696de3e0b7b926d4956fc0249618c85a2c867abdb1dff04414e74537c5c" gracePeriod=30
Nov 02 20:31:22.889941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.889917    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:31:22.890109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.890090    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy" containerID="cri-o://0151cd3a500871eb14b39f47c53a03b00ffd3fb2bf9c85fd36fab38d968d9b90" gracePeriod=30
Nov 02 20:31:22.913662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.913634    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:31:22.913912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:22.913886    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy" containerID="cri-o://63329d22922f2e3810c3210436236a44b8ef3aca5cc2137eb87ec9fb2577638e" gracePeriod=30
Nov 02 20:31:25.108690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.108655    5099 generic.go:356] "Generic (PLEG): container finished" podID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerID="2a99b5cffcb65874f60c17b2b88b386bb1ee72bc24d076fe0a8c8b382a9f9746" exitCode=0
Nov 02 20:31:25.109276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.108690    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" event={"ID":"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a","Type":"ContainerDied","Data":"2a99b5cffcb65874f60c17b2b88b386bb1ee72bc24d076fe0a8c8b382a9f9746"}
Nov 02 20:31:25.184882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.184841    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:31:25.300009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.299970    5099 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:31:25.300311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.300273    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:31:25.323676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.323635    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.323860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.323695    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.323860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.323844    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.323892    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324005    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324095    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324154    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324155    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:25.324193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324173    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:25.324343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324318    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324436    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:25.324554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324485    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\") pod \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\" (UID: \"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a\") "
Nov 02 20:31:25.324619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324595    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.324930    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.325318    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.325345    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.325362    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.325377    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.325427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.325393    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.326220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.326199    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:25.326303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.326240    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:25.326843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.326821    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "kube-api-access-7lgrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:25.327553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.327533    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo" pod "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" (UID: "1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:31:25.426529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.426450    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.426529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.426478    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.426529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.426488    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.426529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.426498    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lgrl\" (UniqueName: \"kubernetes.io/projected/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a-kube-api-access-7lgrl\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:25.782951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.782846    5099 patch_prober.go:29] interesting pod/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:31:25.782951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:25.782913    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:31:26.118412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.118388    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"
Nov 02 20:31:26.118819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.118418    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6" event={"ID":"1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a","Type":"ContainerDied","Data":"1561f1079a6911afed3ed3819b1d8f6b5d07a33cf787fe5f4871cb1f4d6d5740"}
Nov 02 20:31:26.118819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.118483    5099 scope.go:122] "RemoveContainer" containerID="2a99b5cffcb65874f60c17b2b88b386bb1ee72bc24d076fe0a8c8b382a9f9746"
Nov 02 20:31:26.324039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.324002    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:31:26.372614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.372529    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9rtj6"]
Nov 02 20:31:26.736147 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.736122    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:31:26.845428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.845378    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.845708    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.845835    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.845931    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846012    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846073    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846191    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846228    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846296    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\") pod \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\" (UID: \"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9\") "
Nov 02 20:31:26.846649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.846553    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:26.847091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.847065    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.847543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.847522    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:26.847700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.847679    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:26.847880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.847859    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:26.848398 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.848370    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:26.850659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.850624    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:26.850994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.850969    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "kube-api-access-p7plf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:26.851581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.851548    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:26.853037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.853013    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo" pod "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" (UID: "f4f467d9-f03b-41db-8ca3-bb9729a2b6d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948758    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948801    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948812    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948822    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948830    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948840    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948849    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7plf\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-kube-api-access-p7plf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:26.948848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:26.948859    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:27.128073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.128040    5099 generic.go:356] "Generic (PLEG): container finished" podID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerID="dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd" exitCode=0
Nov 02 20:31:27.128557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.128092    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"
Nov 02 20:31:27.128557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.128128    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerDied","Data":"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"}
Nov 02 20:31:27.128557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.128198    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz" event={"ID":"f4f467d9-f03b-41db-8ca3-bb9729a2b6d9","Type":"ContainerDied","Data":"a7008573124d888399641857e1b5d7ef4ff0b4ba32cc877fa8c807ed229df0e4"}
Nov 02 20:31:27.128557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.128234    5099 scope.go:122] "RemoveContainer" containerID="dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"
Nov 02 20:31:27.140965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.140942    5099 scope.go:122] "RemoveContainer" containerID="df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"
Nov 02 20:31:27.156472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.156444    5099 scope.go:122] "RemoveContainer" containerID="dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"
Nov 02 20:31:27.157043 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:27.157017    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd\": container with ID starting with dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd not found: ID does not exist" containerID="dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"
Nov 02 20:31:27.157161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.157055    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd"} err="failed to get container status \"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd\": rpc error: code = NotFound desc = could not find container \"dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd\": container with ID starting with dc82e1ad8f8055c61e84f008b42c1f958cf034be6c14ef9ad0e7d4fc5c0145fd not found: ID does not exist"
Nov 02 20:31:27.157161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.157081    5099 scope.go:122] "RemoveContainer" containerID="df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"
Nov 02 20:31:27.157428 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:27.157410    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b\": container with ID starting with df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b not found: ID does not exist" containerID="df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"
Nov 02 20:31:27.157490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.157437    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b"} err="failed to get container status \"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b\": rpc error: code = NotFound desc = could not find container \"df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b\": container with ID starting with df4de0187e0d21f87744a50ed9a39fbfdd960304a2c1f5cbbc9b5dcf2efae86b not found: ID does not exist"
Nov 02 20:31:27.262339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.262259    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" path="/var/lib/kubelet/pods/1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a/volumes"
Nov 02 20:31:27.288824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.288775    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:31:27.334474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:27.334442    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-429b45-openshift-default-76c7dc5f69-qv2gz"]
Nov 02 20:31:28.144823 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.144721    5099 generic.go:356] "Generic (PLEG): container finished" podID="99642394-2af1-4b51-963e-1415018eb80a" containerID="0151cd3a500871eb14b39f47c53a03b00ffd3fb2bf9c85fd36fab38d968d9b90" exitCode=0
Nov 02 20:31:28.145483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.145251    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" event={"ID":"99642394-2af1-4b51-963e-1415018eb80a","Type":"ContainerDied","Data":"0151cd3a500871eb14b39f47c53a03b00ffd3fb2bf9c85fd36fab38d968d9b90"}
Nov 02 20:31:28.147851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.147250    5099 generic.go:356] "Generic (PLEG): container finished" podID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerID="63329d22922f2e3810c3210436236a44b8ef3aca5cc2137eb87ec9fb2577638e" exitCode=0
Nov 02 20:31:28.147851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.147384    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" event={"ID":"ce9a2dff-eecf-41b6-aeb8-f5d621941e63","Type":"ContainerDied","Data":"63329d22922f2e3810c3210436236a44b8ef3aca5cc2137eb87ec9fb2577638e"}
Nov 02 20:31:28.150515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.149926    5099 generic.go:356] "Generic (PLEG): container finished" podID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerID="23f8a696de3e0b7b926d4956fc0249618c85a2c867abdb1dff04414e74537c5c" exitCode=0
Nov 02 20:31:28.150515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.150037    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerDied","Data":"23f8a696de3e0b7b926d4956fc0249618c85a2c867abdb1dff04414e74537c5c"}
Nov 02 20:31:28.150515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.150079    5099 scope.go:122] "RemoveContainer" containerID="fe92a33b313e7cfb7934e78996f4b8107d0f4a2c03f974e81a4d9bacf04fba9f"
Nov 02 20:31:28.247585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.246997    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:31:28.253845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.252744    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:31:28.312479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.312448    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:31:28.368188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368151    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368278    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368332    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368363    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368396    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368468    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368502    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368593    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368621    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368640    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368655    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368685    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368721    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368761    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368860    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.368889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368888    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\") pod \"99642394-2af1-4b51-963e-1415018eb80a\" (UID: \"99642394-2af1-4b51-963e-1415018eb80a\") "
Nov 02 20:31:28.369268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368912    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.369268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.368950    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.369268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.369006    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.369268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.369097    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.369268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.369104    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\") pod \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\" (UID: \"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485\") "
Nov 02 20:31:28.369621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.369597    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.369808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.369770    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.370029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370009    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.370105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370074    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:28.370382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370363    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.370535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370519    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.370735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370722    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99642394-2af1-4b51-963e-1415018eb80a-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.370866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370853    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.370964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370953    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.371046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.371035    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.371132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.371121    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.371209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370432    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:28.371285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.370750    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.372486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.372458    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.372818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.372794    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:31:28.375633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.375604    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.376613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.376587    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.376815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.376766    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "kube-api-access-dhlms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.376899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.376835    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "kube-api-access-9lll7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.377838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.377772    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:31:28.378124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.378098    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy" pod "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" (UID: "df1d3f3f-71f3-47ce-8eb2-bd9e271e5485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.378304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.378283    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy" pod "99642394-2af1-4b51-963e-1415018eb80a" (UID: "99642394-2af1-4b51-963e-1415018eb80a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.472200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472109    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472206    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472367 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472304    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472423    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.472512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472449    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472504    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472553    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472609    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.472684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472643    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472710    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\") pod \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\" (UID: \"ce9a2dff-eecf-41b6-aeb8-f5d621941e63\") "
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.472938    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473133    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473278    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473489    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473511    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lll7\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-kube-api-access-9lll7\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473526    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99642394-2af1-4b51-963e-1415018eb80a-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473540    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473554    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473567    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473581    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473597    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99642394-2af1-4b51-963e-1415018eb80a-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473610    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473623    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhlms\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-kube-api-access-dhlms\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473638    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473651    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473664    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473677    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473690    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99642394-2af1-4b51-963e-1415018eb80a-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.474165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.473753    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.475810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.475758    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "kube-api-access-td7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.475905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.475880    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:28.476023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.475999    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:31:28.477216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.477193    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy" pod "ce9a2dff-eecf-41b6-aeb8-f5d621941e63" (UID: "ce9a2dff-eecf-41b6-aeb8-f5d621941e63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:31:28.574885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.574853    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.574885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.574882    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td7lf\" (UniqueName: \"kubernetes.io/projected/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-kube-api-access-td7lf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.574885 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.574892    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.575177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.574901    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:28.575177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:28.574910    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9a2dff-eecf-41b6-aeb8-f5d621941e63-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:29.165232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.165143    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"
Nov 02 20:31:29.165707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.165142    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh" event={"ID":"99642394-2af1-4b51-963e-1415018eb80a","Type":"ContainerDied","Data":"66224beed883c35c868e589bb841bcd8165fe700b81a070760e6553e84acb278"}
Nov 02 20:31:29.165707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.165320    5099 scope.go:122] "RemoveContainer" containerID="0151cd3a500871eb14b39f47c53a03b00ffd3fb2bf9c85fd36fab38d968d9b90"
Nov 02 20:31:29.168114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.168078    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv" event={"ID":"ce9a2dff-eecf-41b6-aeb8-f5d621941e63","Type":"ContainerDied","Data":"c22fe53ef60c3e4604094ba7e6b56d4e604ad619b97d32a64b81a0dcc0679940"}
Nov 02 20:31:29.168494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.168170    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"
Nov 02 20:31:29.173212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.172474    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd" event={"ID":"df1d3f3f-71f3-47ce-8eb2-bd9e271e5485","Type":"ContainerDied","Data":"b006fd5cb66ea784b2f87d9d058082f24322ed0772f4f7cb8acc474ffd3958c6"}
Nov 02 20:31:29.173212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.172568    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"
Nov 02 20:31:29.183346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.180915    5099 scope.go:122] "RemoveContainer" containerID="63329d22922f2e3810c3210436236a44b8ef3aca5cc2137eb87ec9fb2577638e"
Nov 02 20:31:29.201320 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.201288    5099 scope.go:122] "RemoveContainer" containerID="23f8a696de3e0b7b926d4956fc0249618c85a2c867abdb1dff04414e74537c5c"
Nov 02 20:31:29.263475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.263440    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" path="/var/lib/kubelet/pods/f4f467d9-f03b-41db-8ca3-bb9729a2b6d9/volumes"
Nov 02 20:31:29.323672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.323635    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:31:29.383543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.383505    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-7cwdh"]
Nov 02 20:31:29.440249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.440165    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:31:29.444365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.444335    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-p7wzv"]
Nov 02 20:31:29.546547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.546511    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:31:29.551449 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:29.551423    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-ck8bd"]
Nov 02 20:31:31.261959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:31.261929    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99642394-2af1-4b51-963e-1415018eb80a" path="/var/lib/kubelet/pods/99642394-2af1-4b51-963e-1415018eb80a/volumes"
Nov 02 20:31:31.262508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:31.262491    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" path="/var/lib/kubelet/pods/ce9a2dff-eecf-41b6-aeb8-f5d621941e63/volumes"
Nov 02 20:31:31.262996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:31.262981    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" path="/var/lib/kubelet/pods/df1d3f3f-71f3-47ce-8eb2-bd9e271e5485/volumes"
Nov 02 20:31:32.216256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:32.216223    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:31:32.331447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:32.331420    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:31:34.739707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:34.739664    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:31:37.216827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.216757    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" podUID="66604f72-a49b-41aa-ba70-26dc77341d00" containerName="payload-processing" containerID="cri-o://75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19" gracePeriod=25
Nov 02 20:31:37.343836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.343093    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" podUID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" containerName="payload-pre-processing" containerID="cri-o://b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317" gracePeriod=25
Nov 02 20:31:37.802512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.802487    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:31:37.806135 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.806114    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:31:37.980618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.980583    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\") pod \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") "
Nov 02 20:31:37.980849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.980724    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\") pod \"66604f72-a49b-41aa-ba70-26dc77341d00\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") "
Nov 02 20:31:37.980947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.980882    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\" (UniqueName: \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\") pod \"66604f72-a49b-41aa-ba70-26dc77341d00\" (UID: \"66604f72-a49b-41aa-ba70-26dc77341d00\") "
Nov 02 20:31:37.980947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.980930    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\" (UniqueName: \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\") pod \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\" (UID: \"c3eab544-7657-4c12-a9da-7cc658a0bfa9\") "
Nov 02 20:31:37.981739 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.981709    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume" pod "c3eab544-7657-4c12-a9da-7cc658a0bfa9" (UID: "c3eab544-7657-4c12-a9da-7cc658a0bfa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:37.981950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.981921    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume" pod "66604f72-a49b-41aa-ba70-26dc77341d00" (UID: "66604f72-a49b-41aa-ba70-26dc77341d00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:37.983090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.983066    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs" pod "c3eab544-7657-4c12-a9da-7cc658a0bfa9" (UID: "c3eab544-7657-4c12-a9da-7cc658a0bfa9"). InnerVolumeSpecName "kube-api-access-d7njs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:37.983879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:37.983861    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw" pod "66604f72-a49b-41aa-ba70-26dc77341d00" (UID: "66604f72-a49b-41aa-ba70-26dc77341d00"). InnerVolumeSpecName "kube-api-access-sbrfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:38.082729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.082645    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7njs\" (UniqueName: \"kubernetes.io/projected/c3eab544-7657-4c12-a9da-7cc658a0bfa9-kube-api-access-d7njs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:38.082729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.082681    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3eab544-7657-4c12-a9da-7cc658a0bfa9-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:38.082729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.082700    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66604f72-a49b-41aa-ba70-26dc77341d00-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:38.082729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.082711    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbrfw\" (UniqueName: \"kubernetes.io/projected/66604f72-a49b-41aa-ba70-26dc77341d00-kube-api-access-sbrfw\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:38.372336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.372302    5099 generic.go:356] "Generic (PLEG): container finished" podID="66604f72-a49b-41aa-ba70-26dc77341d00" containerID="75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19" exitCode=0
Nov 02 20:31:38.372733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.372383    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" event={"ID":"66604f72-a49b-41aa-ba70-26dc77341d00","Type":"ContainerDied","Data":"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"}
Nov 02 20:31:38.372733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.372441    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f" event={"ID":"66604f72-a49b-41aa-ba70-26dc77341d00","Type":"ContainerDied","Data":"56cc9dfb23e41f6189841de46df4769fa640780b86e6d74f0bf254abc3302720"}
Nov 02 20:31:38.372733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.372464    5099 scope.go:122] "RemoveContainer" containerID="75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"
Nov 02 20:31:38.372921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.372903    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"
Nov 02 20:31:38.374869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.374844    5099 generic.go:356] "Generic (PLEG): container finished" podID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" containerID="b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317" exitCode=0
Nov 02 20:31:38.374983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.374883    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" event={"ID":"c3eab544-7657-4c12-a9da-7cc658a0bfa9","Type":"ContainerDied","Data":"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"}
Nov 02 20:31:38.374983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.374953    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6" event={"ID":"c3eab544-7657-4c12-a9da-7cc658a0bfa9","Type":"ContainerDied","Data":"63b2480b988e4916b8a41c11b322a32365e2d949a3c9598049cc32c00c3bc26e"}
Nov 02 20:31:38.375196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.375178    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"
Nov 02 20:31:38.385539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.385519    5099 scope.go:122] "RemoveContainer" containerID="75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"
Nov 02 20:31:38.386042 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:38.386017    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19\": container with ID starting with 75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19 not found: ID does not exist" containerID="75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"
Nov 02 20:31:38.386177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.386156    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19"} err="failed to get container status \"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19\": rpc error: code = NotFound desc = could not find container \"75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19\": container with ID starting with 75a0e3fba4e2a8b1b07539a88ca3b865e147388b5cecd5f0200c0ee9a20f8e19 not found: ID does not exist"
Nov 02 20:31:38.386228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.386181    5099 scope.go:122] "RemoveContainer" containerID="b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"
Nov 02 20:31:38.397117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.397092    5099 scope.go:122] "RemoveContainer" containerID="b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"
Nov 02 20:31:38.397399 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:38.397328    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317\": container with ID starting with b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317 not found: ID does not exist" containerID="b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"
Nov 02 20:31:38.397445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.397408    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317"} err="failed to get container status \"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317\": rpc error: code = NotFound desc = could not find container \"b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317\": container with ID starting with b8b82f7c8b72bc659f2b8561ede70bd0a86229695349829e2b926f7d4b20a317 not found: ID does not exist"
Nov 02 20:31:38.533207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.533167    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:31:38.577572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.577538    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8c5e8e-75db5f47d5-9m99f"]
Nov 02 20:31:38.681256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.681081    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:31:38.685985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:38.685957    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c5e8e-6f8dcc987f-kpnv6"]
Nov 02 20:31:39.291527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:39.291490    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66604f72-a49b-41aa-ba70-26dc77341d00" path="/var/lib/kubelet/pods/66604f72-a49b-41aa-ba70-26dc77341d00/volumes"
Nov 02 20:31:39.292430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:39.292401    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" path="/var/lib/kubelet/pods/c3eab544-7657-4c12-a9da-7cc658a0bfa9/volumes"
Nov 02 20:31:39.754895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:39.754852    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" podUID="79956486-4b68-4095-8d4d-1ef90c39fcfc" containerName="payload-processing" containerID="cri-o://6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556" gracePeriod=25
Nov 02 20:31:40.099416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.099387    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:31:40.222754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.221715    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\" (UniqueName: \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\") pod \"79956486-4b68-4095-8d4d-1ef90c39fcfc\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") "
Nov 02 20:31:40.222754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.221999    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\") pod \"79956486-4b68-4095-8d4d-1ef90c39fcfc\" (UID: \"79956486-4b68-4095-8d4d-1ef90c39fcfc\") "
Nov 02 20:31:40.223691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.223658    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume" pod "79956486-4b68-4095-8d4d-1ef90c39fcfc" (UID: "79956486-4b68-4095-8d4d-1ef90c39fcfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:31:40.226934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.226898    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n" pod "79956486-4b68-4095-8d4d-1ef90c39fcfc" (UID: "79956486-4b68-4095-8d4d-1ef90c39fcfc"). InnerVolumeSpecName "kube-api-access-wn69n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:31:40.324520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.324446    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn69n\" (UniqueName: \"kubernetes.io/projected/79956486-4b68-4095-8d4d-1ef90c39fcfc-kube-api-access-wn69n\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:40.324722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.324709    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79956486-4b68-4095-8d4d-1ef90c39fcfc-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:31:40.396250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.396216    5099 generic.go:356] "Generic (PLEG): container finished" podID="79956486-4b68-4095-8d4d-1ef90c39fcfc" containerID="6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556" exitCode=0
Nov 02 20:31:40.396455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.396296    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" event={"ID":"79956486-4b68-4095-8d4d-1ef90c39fcfc","Type":"ContainerDied","Data":"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"}
Nov 02 20:31:40.396455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.396351    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r" event={"ID":"79956486-4b68-4095-8d4d-1ef90c39fcfc","Type":"ContainerDied","Data":"65267b44e264be061be1f176db4fe75d579f5bbb9aae4c9b607ca2db9b06b39c"}
Nov 02 20:31:40.396455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.396375    5099 scope.go:122] "RemoveContainer" containerID="6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"
Nov 02 20:31:40.396708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.396688    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"
Nov 02 20:31:40.409565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.409547    5099 scope.go:122] "RemoveContainer" containerID="6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"
Nov 02 20:31:40.409857 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:40.409838    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556\": container with ID starting with 6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556 not found: ID does not exist" containerID="6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"
Nov 02 20:31:40.409918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.409864    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556"} err="failed to get container status \"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556\": rpc error: code = NotFound desc = could not find container \"6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556\": container with ID starting with 6c5eff7f7667f8920f7a13c5265cf5e2c5bc5abb827edd1ab9a0e1c8b342d556 not found: ID does not exist"
Nov 02 20:31:40.576627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.576543    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:31:40.580486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:40.580465    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-be1348-556d55cc7-tzn5r"]
Nov 02 20:31:41.278223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:41.278181    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79956486-4b68-4095-8d4d-1ef90c39fcfc" path="/var/lib/kubelet/pods/79956486-4b68-4095-8d4d-1ef90c39fcfc/volumes"
Nov 02 20:31:45.718651 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:45.718615    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:31:46.096386 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:31:46.096289    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:31:47.517425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.517388    5099 generic.go:356] "Generic (PLEG): container finished" podID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerID="fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20" exitCode=0
Nov 02 20:31:47.517898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.517427    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerDied","Data":"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"}
Nov 02 20:31:47.517955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.517911    5099 scope.go:122] "RemoveContainer" containerID="fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"
Nov 02 20:31:47.519551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.519524    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:47.519635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.519592    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:47.519673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:47.519640    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:31:48.527615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:48.527581    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerStarted","Data":"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"}
Nov 02 20:31:48.886217 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:48.886173    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:31:48.886465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:48.886419    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:48.886571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:48.886517    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:49.885519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:49.885480    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:49.886046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:49.885532    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:50.885547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:50.885507    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:50.886069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:50.885557    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:51.886274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:51.886226    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:51.886768 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:51.886294    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:51.898591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:51.898568    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:31:51.924397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:51.924371    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:31:52.886456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:52.886411    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:52.886926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:52.886485    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:53.706587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:53.706552    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:31:53.886396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:53.886360    5099 patch_prober.go:29] interesting pod/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 02 20:31:53.886666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:53.886638    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 02 20:31:54.890328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:54.890297    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:31:54.891437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:31:54.891415    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:32:00.740475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:00.740427    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main" containerID="cri-o://11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30" gracePeriod=45
Nov 02 20:32:01.013932 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.013904    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:32:01.136349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136315    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\" (UniqueName: \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.136554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136485    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.136634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136564    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.136634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136607    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.136752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136681    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.136862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136767    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:01.136862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.136817    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\") pod \"c0cffea3-f787-48a0-ab5e-d02f85984911\" (UID: \"c0cffea3-f787-48a0-ab5e-d02f85984911\") "
Nov 02 20:32:01.137364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.137026    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:01.137364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.137278    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:01.137602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.137583    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.137672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.137608    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.137672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.137624    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.139359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.139320    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "kube-api-access-w8cl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:01.139479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.139444    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:32:01.139622 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.139600    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm" pod "c0cffea3-f787-48a0-ab5e-d02f85984911" (UID: "c0cffea3-f787-48a0-ab5e-d02f85984911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:32:01.239284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.239248    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cffea3-f787-48a0-ab5e-d02f85984911-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.239284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.239283    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8cl6\" (UniqueName: \"kubernetes.io/projected/c0cffea3-f787-48a0-ab5e-d02f85984911-kube-api-access-w8cl6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.239533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.239294    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cffea3-f787-48a0-ab5e-d02f85984911-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:01.721095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.721059    5099 generic.go:356] "Generic (PLEG): container finished" podID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerID="11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30" exitCode=0
Nov 02 20:32:01.721277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.721122    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"
Nov 02 20:32:01.721277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.721140    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" event={"ID":"c0cffea3-f787-48a0-ab5e-d02f85984911","Type":"ContainerDied","Data":"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"}
Nov 02 20:32:01.721277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.721185    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r" event={"ID":"c0cffea3-f787-48a0-ab5e-d02f85984911","Type":"ContainerDied","Data":"9cd6a90e2449ae317b1ead008684ab60045d037e16a69b0561b79bb4db02556a"}
Nov 02 20:32:01.721277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.721206    5099 scope.go:122] "RemoveContainer" containerID="11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"
Nov 02 20:32:01.731718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.731696    5099 scope.go:122] "RemoveContainer" containerID="11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"
Nov 02 20:32:01.732045 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:01.732018    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30\": container with ID starting with 11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30 not found: ID does not exist" containerID="11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"
Nov 02 20:32:01.732107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.732056    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30"} err="failed to get container status \"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30\": rpc error: code = NotFound desc = could not find container \"11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30\": container with ID starting with 11e92a90f850fe03f2599733593102bea3ea3deb5d6598bfc2f1171ec7416b30 not found: ID does not exist"
Nov 02 20:32:01.877420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.877388    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:32:01.924040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:01.924001    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-w2-9fcedd-kserve-5886f586fb-szd9r"]
Nov 02 20:32:03.261748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:03.261716    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" path="/var/lib/kubelet/pods/c0cffea3-f787-48a0-ab5e-d02f85984911/volumes"
Nov 02 20:32:04.668728 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:04.668687    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:32:04.791065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:04.786831    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:32:09.669364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:09.669265    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" podUID="250118d2-58de-4f36-82b4-85528f5e6d1c" containerName="payload-processing" containerID="cri-o://8ee90f21697dfde0e5e2d9b6150e89aad1f3e8980bd96aad94b3776e781c0e29" gracePeriod=25
Nov 02 20:32:09.787451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:09.787414    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" podUID="eca444a5-093b-4737-8949-7ec6a41941b7" containerName="payload-pre-processing" containerID="cri-o://1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7" gracePeriod=25
Nov 02 20:32:09.789997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:09.789970    5099 generic.go:356] "Generic (PLEG): container finished" podID="250118d2-58de-4f36-82b4-85528f5e6d1c" containerID="8ee90f21697dfde0e5e2d9b6150e89aad1f3e8980bd96aad94b3776e781c0e29" exitCode=0
Nov 02 20:32:09.790119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:09.790049    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" event={"ID":"250118d2-58de-4f36-82b4-85528f5e6d1c","Type":"ContainerDied","Data":"8ee90f21697dfde0e5e2d9b6150e89aad1f3e8980bd96aad94b3776e781c0e29"}
Nov 02 20:32:09.990637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:09.989832    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:32:10.087947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.087912    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\") pod \"250118d2-58de-4f36-82b4-85528f5e6d1c\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") "
Nov 02 20:32:10.088288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.088267    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume" pod "250118d2-58de-4f36-82b4-85528f5e6d1c" (UID: "250118d2-58de-4f36-82b4-85528f5e6d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:10.088529 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.088511    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\" (UniqueName: \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\") pod \"250118d2-58de-4f36-82b4-85528f5e6d1c\" (UID: \"250118d2-58de-4f36-82b4-85528f5e6d1c\") "
Nov 02 20:32:10.088911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.088893    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250118d2-58de-4f36-82b4-85528f5e6d1c-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:10.090769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.090741    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds" pod "250118d2-58de-4f36-82b4-85528f5e6d1c" (UID: "250118d2-58de-4f36-82b4-85528f5e6d1c"). InnerVolumeSpecName "kube-api-access-ppcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:10.092073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.092053    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:32:10.189800 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.189750    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\") pod \"eca444a5-093b-4737-8949-7ec6a41941b7\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") "
Nov 02 20:32:10.190054 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.189972    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\" (UniqueName: \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\") pod \"eca444a5-093b-4737-8949-7ec6a41941b7\" (UID: \"eca444a5-093b-4737-8949-7ec6a41941b7\") "
Nov 02 20:32:10.190190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.190164    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume" pod "eca444a5-093b-4737-8949-7ec6a41941b7" (UID: "eca444a5-093b-4737-8949-7ec6a41941b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:10.190492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.190465    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eca444a5-093b-4737-8949-7ec6a41941b7-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:10.190492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.190484    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcds\" (UniqueName: \"kubernetes.io/projected/250118d2-58de-4f36-82b4-85528f5e6d1c-kube-api-access-ppcds\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:10.192119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.192098    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk" pod "eca444a5-093b-4737-8949-7ec6a41941b7" (UID: "eca444a5-093b-4737-8949-7ec6a41941b7"). InnerVolumeSpecName "kube-api-access-jkpfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:10.316256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.304426    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpfk\" (UniqueName: \"kubernetes.io/projected/eca444a5-093b-4737-8949-7ec6a41941b7-kube-api-access-jkpfk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:10.799606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.799568    5099 generic.go:356] "Generic (PLEG): container finished" podID="eca444a5-093b-4737-8949-7ec6a41941b7" containerID="1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7" exitCode=0
Nov 02 20:32:10.800107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.799617    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"
Nov 02 20:32:10.800107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.799653    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" event={"ID":"eca444a5-093b-4737-8949-7ec6a41941b7","Type":"ContainerDied","Data":"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"}
Nov 02 20:32:10.800107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.799700    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd" event={"ID":"eca444a5-093b-4737-8949-7ec6a41941b7","Type":"ContainerDied","Data":"40791847bc5230766ed5fdebbbc1a87dc3292d1d78ced238481e70e1fcf9aaea"}
Nov 02 20:32:10.800107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.799726    5099 scope.go:122] "RemoveContainer" containerID="1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"
Nov 02 20:32:10.801410 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.801394    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"
Nov 02 20:32:10.801499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.801394    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6" event={"ID":"250118d2-58de-4f36-82b4-85528f5e6d1c","Type":"ContainerDied","Data":"f12ee72922306292912aef62906addbe03ad6f0af47ab44d746ffaef87fb911d"}
Nov 02 20:32:10.813274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.813253    5099 scope.go:122] "RemoveContainer" containerID="1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"
Nov 02 20:32:10.813586 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:10.813568    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7\": container with ID starting with 1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7 not found: ID does not exist" containerID="1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"
Nov 02 20:32:10.813660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.813591    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7"} err="failed to get container status \"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7\": rpc error: code = NotFound desc = could not find container \"1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7\": container with ID starting with 1f96eb347c8e119be4f1fb754fef5c45894f24a09481937a6fc64d2c8d213ec7 not found: ID does not exist"
Nov 02 20:32:10.813660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.813609    5099 scope.go:122] "RemoveContainer" containerID="8ee90f21697dfde0e5e2d9b6150e89aad1f3e8980bd96aad94b3776e781c0e29"
Nov 02 20:32:10.957878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:10.957764    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:32:11.003283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:11.003250    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a51b3f-5f5fb86cb7-4czcd"]
Nov 02 20:32:11.117504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:11.117463    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:32:11.121460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:11.121431    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a51b3f-b449df449-jztp6"]
Nov 02 20:32:11.261501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:11.261410    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250118d2-58de-4f36-82b4-85528f5e6d1c" path="/var/lib/kubelet/pods/250118d2-58de-4f36-82b4-85528f5e6d1c/volumes"
Nov 02 20:32:11.261907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:11.261890    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca444a5-093b-4737-8949-7ec6a41941b7" path="/var/lib/kubelet/pods/eca444a5-093b-4737-8949-7ec6a41941b7/volumes"
Nov 02 20:32:16.095689 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:16.095649    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:32:26.529011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:26.528974    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:32:26.670929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:26.670878    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:32:31.530108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:31.530005    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" podUID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" containerName="payload-processing" containerID="cri-o://c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536" gracePeriod=25
Nov 02 20:32:31.672020 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:31.671951    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" podUID="bd4fa185-ad29-4705-beec-4dcf27679c0d" containerName="payload-pre-processing" containerID="cri-o://464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9" gracePeriod=25
Nov 02 20:32:31.846883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:31.846852    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:32:31.983452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:31.983424    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:32:32.007625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.007582    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\" (UniqueName: \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\") pod \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") "
Nov 02 20:32:32.007796 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.007636    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\") pod \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\" (UID: \"52fcd0b0-0af2-44b0-aed0-5a959d82623a\") "
Nov 02 20:32:32.008242 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.008125    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume" pod "52fcd0b0-0af2-44b0-aed0-5a959d82623a" (UID: "52fcd0b0-0af2-44b0-aed0-5a959d82623a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:32.008402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.008378    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fcd0b0-0af2-44b0-aed0-5a959d82623a-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:32.010347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.010320    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866" pod "52fcd0b0-0af2-44b0-aed0-5a959d82623a" (UID: "52fcd0b0-0af2-44b0-aed0-5a959d82623a"). InnerVolumeSpecName "kube-api-access-kq866". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:32.109799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.109751    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\" (UniqueName: \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\") pod \"bd4fa185-ad29-4705-beec-4dcf27679c0d\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") "
Nov 02 20:32:32.109996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.109870    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\") pod \"bd4fa185-ad29-4705-beec-4dcf27679c0d\" (UID: \"bd4fa185-ad29-4705-beec-4dcf27679c0d\") "
Nov 02 20:32:32.110588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.110557    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume" pod "bd4fa185-ad29-4705-beec-4dcf27679c0d" (UID: "bd4fa185-ad29-4705-beec-4dcf27679c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:32.110588 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.110586    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq866\" (UniqueName: \"kubernetes.io/projected/52fcd0b0-0af2-44b0-aed0-5a959d82623a-kube-api-access-kq866\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:32.112495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.112472    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t" pod "bd4fa185-ad29-4705-beec-4dcf27679c0d" (UID: "bd4fa185-ad29-4705-beec-4dcf27679c0d"). InnerVolumeSpecName "kube-api-access-wf56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:32.139933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.139896    5099 generic.go:356] "Generic (PLEG): container finished" podID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" containerID="c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536" exitCode=0
Nov 02 20:32:32.140124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.139951    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"
Nov 02 20:32:32.140124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.139997    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" event={"ID":"52fcd0b0-0af2-44b0-aed0-5a959d82623a","Type":"ContainerDied","Data":"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"}
Nov 02 20:32:32.140124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.140075    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt" event={"ID":"52fcd0b0-0af2-44b0-aed0-5a959d82623a","Type":"ContainerDied","Data":"8535ba28f7b9ca724c47b8386b8914c44511c8b86863986187375fc195cf8df3"}
Nov 02 20:32:32.140124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.140109    5099 scope.go:122] "RemoveContainer" containerID="c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"
Nov 02 20:32:32.142844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.142045    5099 generic.go:356] "Generic (PLEG): container finished" podID="bd4fa185-ad29-4705-beec-4dcf27679c0d" containerID="464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9" exitCode=0
Nov 02 20:32:32.142844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.142201    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" event={"ID":"bd4fa185-ad29-4705-beec-4dcf27679c0d","Type":"ContainerDied","Data":"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"}
Nov 02 20:32:32.142844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.142244    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5" event={"ID":"bd4fa185-ad29-4705-beec-4dcf27679c0d","Type":"ContainerDied","Data":"dab4879f77f2cd73b4bee8a4248228360df7eb82d3b12516657fc6c586915950"}
Nov 02 20:32:32.142844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.142318    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"
Nov 02 20:32:32.152665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.152626    5099 generic.go:356] "Generic (PLEG): container finished" podID="7b58d473-99cb-4c46-bb41-77c27a057533" containerID="ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909" exitCode=0
Nov 02 20:32:32.152835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.152700    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerDied","Data":"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"}
Nov 02 20:32:32.152999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.152966    5099 scope.go:122] "RemoveContainer" containerID="c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"
Nov 02 20:32:32.153483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.153224    5099 scope.go:122] "RemoveContainer" containerID="ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"
Nov 02 20:32:32.153483 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:32.153374    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536\": container with ID starting with c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536 not found: ID does not exist" containerID="c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"
Nov 02 20:32:32.153483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.153400    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536"} err="failed to get container status \"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536\": rpc error: code = NotFound desc = could not find container \"c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536\": container with ID starting with c9b254fad62f6f844b62994071949f8b3c417ca6d2e6337eed34ac8602d7b536 not found: ID does not exist"
Nov 02 20:32:32.153483 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.153420    5099 scope.go:122] "RemoveContainer" containerID="464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"
Nov 02 20:32:32.155031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.154943    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:32:32.155031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.155024    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:32:32.155167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.155062    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:32:32.166285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.166261    5099 scope.go:122] "RemoveContainer" containerID="464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"
Nov 02 20:32:32.166580 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:32.166560    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9\": container with ID starting with 464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9 not found: ID does not exist" containerID="464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"
Nov 02 20:32:32.166666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.166586    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9"} err="failed to get container status \"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9\": rpc error: code = NotFound desc = could not find container \"464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9\": container with ID starting with 464f21765a22570be9be8725e91b22d036a5702558f4cb4dc50114369463f7f9 not found: ID does not exist"
Nov 02 20:32:32.212109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.212083    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf56t\" (UniqueName: \"kubernetes.io/projected/bd4fa185-ad29-4705-beec-4dcf27679c0d-kube-api-access-wf56t\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:32.212109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.212107    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd4fa185-ad29-4705-beec-4dcf27679c0d-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:32.295796 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.295744    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:32:32.299326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.299298    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ab501-5d86bbf767-c8kb5"]
Nov 02 20:32:32.515073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.514982    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:32:32.522235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:32.522198    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ab501-76655c59df-s5rkt"]
Nov 02 20:32:33.177523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:33.177480    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerStarted","Data":"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"}
Nov 02 20:32:33.273447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:33.273409    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" path="/var/lib/kubelet/pods/52fcd0b0-0af2-44b0-aed0-5a959d82623a/volumes"
Nov 02 20:32:33.273918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:33.273903    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4fa185-ad29-4705-beec-4dcf27679c0d" path="/var/lib/kubelet/pods/bd4fa185-ad29-4705-beec-4dcf27679c0d/volumes"
Nov 02 20:32:34.132361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:34.132320    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:32:34.132544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:34.132493    5099 patch_prober.go:29] interesting pod/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 20:32:34.132601 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:34.132552    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 20:32:35.132296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:35.132260    5099 patch_prober.go:29] interesting pod/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 20:32:35.132841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:35.132335    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 20:32:36.132560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:36.132522    5099 patch_prober.go:29] interesting pod/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 20:32:36.133112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:36.132595    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 20:32:37.132391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:37.132347    5099 patch_prober.go:29] interesting pod/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 20:32:37.132585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:37.132412    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 20:32:38.132858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:38.132816    5099 patch_prober.go:29] interesting pod/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 20:32:38.133321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:38.132895    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 20:32:39.138170 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:39.138124    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:32:39.138639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:39.138602    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:32:39.139369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:39.139345    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:32:46.094127 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:46.094093    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:32:48.654203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:48.654163    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:32:48.654642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:48.654367    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" podUID="fcf57063-a180-4370-8794-37e58ffbe2e7" containerName="maas-api" containerID="cri-o://1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f" gracePeriod=30
Nov 02 20:32:48.995638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:48.995606    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:32:49.099898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.096221    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\" (UniqueName: \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\") pod \"fcf57063-a180-4370-8794-37e58ffbe2e7\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") "
Nov 02 20:32:49.099898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.096306    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") pod \"fcf57063-a180-4370-8794-37e58ffbe2e7\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") "
Nov 02 20:32:49.099898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.096465    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\") pod \"fcf57063-a180-4370-8794-37e58ffbe2e7\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") "
Nov 02 20:32:49.099898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.096570    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\") pod \"fcf57063-a180-4370-8794-37e58ffbe2e7\" (UID: \"fcf57063-a180-4370-8794-37e58ffbe2e7\") "
Nov 02 20:32:49.101298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.101250    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca" pod "fcf57063-a180-4370-8794-37e58ffbe2e7" (UID: "fcf57063-a180-4370-8794-37e58ffbe2e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:32:49.119129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.118468    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs" pod "fcf57063-a180-4370-8794-37e58ffbe2e7" (UID: "fcf57063-a180-4370-8794-37e58ffbe2e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:32:49.133970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.133928    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls" pod "fcf57063-a180-4370-8794-37e58ffbe2e7" (UID: "fcf57063-a180-4370-8794-37e58ffbe2e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:32:49.144223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.144178    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x" pod "fcf57063-a180-4370-8794-37e58ffbe2e7" (UID: "fcf57063-a180-4370-8794-37e58ffbe2e7"). InnerVolumeSpecName "kube-api-access-p7s7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:32:49.199023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.198985    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:49.199023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.199022    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7s7x\" (UniqueName: \"kubernetes.io/projected/fcf57063-a180-4370-8794-37e58ffbe2e7-kube-api-access-p7s7x\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:49.199240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.199040    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcf57063-a180-4370-8794-37e58ffbe2e7-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:49.199240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.199054    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf57063-a180-4370-8794-37e58ffbe2e7-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:32:49.334089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.334053    5099 generic.go:356] "Generic (PLEG): container finished" podID="fcf57063-a180-4370-8794-37e58ffbe2e7" containerID="1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f" exitCode=0
Nov 02 20:32:49.334279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.334102    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"
Nov 02 20:32:49.334279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.334135    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" event={"ID":"fcf57063-a180-4370-8794-37e58ffbe2e7","Type":"ContainerDied","Data":"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"}
Nov 02 20:32:49.334279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.334192    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g" event={"ID":"fcf57063-a180-4370-8794-37e58ffbe2e7","Type":"ContainerDied","Data":"3fe1e1de891ef3ee10c8939ac75e847b5357b9c9a82c0c25c38558831f44e0d3"}
Nov 02 20:32:49.334279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.334216    5099 scope.go:122] "RemoveContainer" containerID="1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"
Nov 02 20:32:49.344920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.344892    5099 scope.go:122] "RemoveContainer" containerID="1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"
Nov 02 20:32:49.345179 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:32:49.345152    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f\": container with ID starting with 1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f not found: ID does not exist" containerID="1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"
Nov 02 20:32:49.345253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.345190    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f"} err="failed to get container status \"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f\": rpc error: code = NotFound desc = could not find container \"1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f\": container with ID starting with 1d1176bc63907caea746056a722f60036e83bf3129cc6e3e93c927b5e324d26f not found: ID does not exist"
Nov 02 20:32:49.490623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.490589    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:32:49.495747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:49.495713    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c5e8e-67dd7cc858-fcl9g"]
Nov 02 20:32:51.259369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:51.259336    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf57063-a180-4370-8794-37e58ffbe2e7" path="/var/lib/kubelet/pods/fcf57063-a180-4370-8794-37e58ffbe2e7/volumes"
Nov 02 20:32:58.996282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.996242    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:32:58.997004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.996988    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main"
Nov 02 20:32:58.997004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997005    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997028    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997034    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997043    5099 cpu_manager.go:414] "containerMap: removing container" podUID="66604f72-a49b-41aa-ba70-26dc77341d00" containerName="payload-processing"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997048    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66604f72-a49b-41aa-ba70-26dc77341d00" containerName="payload-processing"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997055    5099 cpu_manager.go:414] "containerMap: removing container" podUID="fcf57063-a180-4370-8794-37e58ffbe2e7" containerName="maas-api"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997060    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf57063-a180-4370-8794-37e58ffbe2e7" containerName="maas-api"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997068    5099 cpu_manager.go:414] "containerMap: removing container" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997073    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997081    5099 cpu_manager.go:414] "containerMap: removing container" podUID="79956486-4b68-4095-8d4d-1ef90c39fcfc" containerName="payload-processing"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997086    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79956486-4b68-4095-8d4d-1ef90c39fcfc" containerName="payload-processing"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997097    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy"
Nov 02 20:32:58.997101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997101    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997109    5099 cpu_manager.go:414] "containerMap: removing container" podUID="bd4fa185-ad29-4705-beec-4dcf27679c0d" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997114    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4fa185-ad29-4705-beec-4dcf27679c0d" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997131    5099 cpu_manager.go:414] "containerMap: removing container" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997135    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997145    5099 cpu_manager.go:414] "containerMap: removing container" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997149    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997160    5099 cpu_manager.go:414] "containerMap: removing container" podUID="eca444a5-093b-4737-8949-7ec6a41941b7" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997165    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca444a5-093b-4737-8949-7ec6a41941b7" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997172    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997178    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997186    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997191    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997199    5099 cpu_manager.go:414] "containerMap: removing container" podUID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997203    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997212    5099 cpu_manager.go:414] "containerMap: removing container" podUID="250118d2-58de-4f36-82b4-85528f5e6d1c" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997217    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250118d2-58de-4f36-82b4-85528f5e6d1c" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997224    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997227    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997235    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997239    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997340    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="66604f72-a49b-41aa-ba70-26dc77341d00" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997348    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf57063-a180-4370-8794-37e58ffbe2e7" containerName="maas-api"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997354    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997359    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79956486-4b68-4095-8d4d-1ef90c39fcfc" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997369    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9a2dff-eecf-41b6-aeb8-f5d621941e63" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997375    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4fa185-ad29-4705-beec-4dcf27679c0d" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997382    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="99642394-2af1-4b51-963e-1415018eb80a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997390    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca444a5-093b-4737-8949-7ec6a41941b7" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997397    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8aa72e-c589-4c64-a7e3-356e2c0d6b5a" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997402    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997407    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fcd0b0-0af2-44b0-aed0-5a959d82623a" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997413    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="250118d2-58de-4f36-82b4-85528f5e6d1c" containerName="payload-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997419    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f467d9-f03b-41db-8ca3-bb9729a2b6d9" containerName="istio-proxy"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997425    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3eab544-7657-4c12-a9da-7cc658a0bfa9" containerName="payload-pre-processing"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997433    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6319c39-77fd-4bdb-aaa1-a70cb4551c87" containerName="main"
Nov 02 20:32:58.997494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:58.997441    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cffea3-f787-48a0-ab5e-d02f85984911" containerName="main"
Nov 02 20:32:59.002915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.002895    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.061598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.061564    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0d9ec0-openshift-default-dockercfg-69t5l\""
Nov 02 20:32:59.074058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074022    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074089    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074173    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074196    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074226    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074260    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7r6\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074289    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074415    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.074492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.074438    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.115095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.115053    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185220    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185273    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185322    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185361    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7r6\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185388    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185554    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185589    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185647    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.185811 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.185716    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.187625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.186987    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.203166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.197534    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.203166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.198161    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.203166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.198394    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.203166 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.198920    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.204391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.204367    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.209358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.209332    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.253869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.253746    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.254153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.254128    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7r6\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\") pod \"e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") " pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.317185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.317148    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:32:59.750245 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.750206    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:32:59.750977 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:32:59.750944    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2328bc6_9fe4_470d_bd9e_3f494dcc1c69.slice/crio-9da3408a94c0b8d434549f7c5b4c7f58904eb2a1c60721f90bf0f76ed6c5a0cf WatchSource:0}: Error finding container 9da3408a94c0b8d434549f7c5b4c7f58904eb2a1c60721f90bf0f76ed6c5a0cf: Status 404 returned error can't find the container with id 9da3408a94c0b8d434549f7c5b4c7f58904eb2a1c60721f90bf0f76ed6c5a0cf
Nov 02 20:32:59.754112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.754081    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:32:59.754194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.754146    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:32:59.754194 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:32:59.754175    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:00.434855 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:00.434818    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerStarted","Data":"6af8800b8c098f31670f1eb0da097daffd5df138a3088641c7fe0fba220b605e"}
Nov 02 20:33:00.435254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:00.434876    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerStarted","Data":"9da3408a94c0b8d434549f7c5b4c7f58904eb2a1c60721f90bf0f76ed6c5a0cf"}
Nov 02 20:33:00.550650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:00.550590    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podStartSLOduration=2.550571937 podStartE2EDuration="2.550571937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:00.544215836 +0000 UTC m=+2469.418486725" watchObservedRunningTime="2026-11-02 20:33:00.550571937 +0000 UTC m=+2469.424842826"
Nov 02 20:33:01.317907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:01.317872    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:33:01.323469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:01.323443    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:33:01.443717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:01.443680    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:33:01.444907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:01.444880    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:33:09.313685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.313649    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:33:09.314841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.314820    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1d3f3f-71f3-47ce-8eb2-bd9e271e5485" containerName="istio-proxy"
Nov 02 20:33:09.319360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.319341    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:33:09.319547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.319456    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.369108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.369061    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0d9ec0\""
Nov 02 20:33:09.369292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.369236    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0d9ec0-dockercfg-crvzv\""
Nov 02 20:33:09.506197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.506168    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dmm\" (UniqueName: \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.506402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.506229    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.607426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.607338    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dmm\" (UniqueName: \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.607426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.607393    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.608102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.608082    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.664036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.664006    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dmm\" (UniqueName: \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\") pod \"payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:09.935976 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:09.935939    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:10.293947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.293862    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:33:10.298766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.298746    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336025    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllpj\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336079    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336338    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336389    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336491    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336545    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336723    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336751    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.336960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.336918    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.346388 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:33:10.346364    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7295f041_0cc0_424b_8de6_707167639992.slice/crio-ac99f4b7c3dea5e68439d08ff3e3c5f4ef8da8546bfbcffd29bce2a0b53b1791 WatchSource:0}: Error finding container ac99f4b7c3dea5e68439d08ff3e3c5f4ef8da8546bfbcffd29bce2a0b53b1791: Status 404 returned error can't find the container with id ac99f4b7c3dea5e68439d08ff3e3c5f4ef8da8546bfbcffd29bce2a0b53b1791
Nov 02 20:33:10.346826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.346801    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:33:10.382642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.382616    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b1839-openshift-default-dockercfg-rh2qm\""
Nov 02 20:33:10.395879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.391083    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:33:10.438592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438560    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438613    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438682    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438721    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438767    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438829    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.438980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.438936    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.439140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.439019    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllpj\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.439140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.439048    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.439490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.439467    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.439751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.439733    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.440988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.440963    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.441229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.441208    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.441470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.441449    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.456710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.455729    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.459287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.459239    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.505167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.503811    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllpj\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.544477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.544394    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\") pod \"e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") " pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.563091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.563047    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" event={"ID":"7295f041-0cc0-424b-8de6-707167639992","Type":"ContainerStarted","Data":"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"}
Nov 02 20:33:10.563305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.563115    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" event={"ID":"7295f041-0cc0-424b-8de6-707167639992","Type":"ContainerStarted","Data":"ac99f4b7c3dea5e68439d08ff3e3c5f4ef8da8546bfbcffd29bce2a0b53b1791"}
Nov 02 20:33:10.563721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.563670    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:10.613836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.613798    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:10.681978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:10.681904    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" podStartSLOduration=1.681880616 podStartE2EDuration="1.681880616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:10.674634363 +0000 UTC m=+2479.548905252" watchObservedRunningTime="2026-11-02 20:33:10.681880616 +0000 UTC m=+2479.556151517"
Nov 02 20:33:11.021096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.021066    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:33:11.021741 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:33:11.021710    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1611b8bf_6b4e_437f_8fe3_ca6681f22722.slice/crio-0e50bce301b37455687e832d0efac88f2f347d1360f98d6cf5a932c014181263 WatchSource:0}: Error finding container 0e50bce301b37455687e832d0efac88f2f347d1360f98d6cf5a932c014181263: Status 404 returned error can't find the container with id 0e50bce301b37455687e832d0efac88f2f347d1360f98d6cf5a932c014181263
Nov 02 20:33:11.024695 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.024662    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:11.024830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.024727    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:11.024830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.024755    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:11.573279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.573243    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerStarted","Data":"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"}
Nov 02 20:33:11.573696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.573306    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerStarted","Data":"0e50bce301b37455687e832d0efac88f2f347d1360f98d6cf5a932c014181263"}
Nov 02 20:33:11.614564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.614531    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:11.686948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:11.686900    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podStartSLOduration=1.6868860460000001 podStartE2EDuration="1.686886046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:11.683479321 +0000 UTC m=+2480.557750225" watchObservedRunningTime="2026-11-02 20:33:11.686886046 +0000 UTC m=+2480.561156934"
Nov 02 20:33:12.620152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:12.620121    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:13.588851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:13.588821    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:13.590021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:13.589995    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:33:16.095209 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:16.095167    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:33:18.601657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:18.601614    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:33:18.602652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:18.602596    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" podUID="3739c5da-cc9e-4017-a820-143420717b78" containerName="maas-api" containerID="cri-o://98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6" gracePeriod=30
Nov 02 20:33:19.080696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.080663    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:33:19.167874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.167324    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\") pod \"3739c5da-cc9e-4017-a820-143420717b78\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") "
Nov 02 20:33:19.167874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.167440    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\" (UniqueName: \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\") pod \"3739c5da-cc9e-4017-a820-143420717b78\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") "
Nov 02 20:33:19.167874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.167527    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\") pod \"3739c5da-cc9e-4017-a820-143420717b78\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") "
Nov 02 20:33:19.167874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.167706    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") pod \"3739c5da-cc9e-4017-a820-143420717b78\" (UID: \"3739c5da-cc9e-4017-a820-143420717b78\") "
Nov 02 20:33:19.171802 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.169633    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca" pod "3739c5da-cc9e-4017-a820-143420717b78" (UID: "3739c5da-cc9e-4017-a820-143420717b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:19.175914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.175864    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs" pod "3739c5da-cc9e-4017-a820-143420717b78" (UID: "3739c5da-cc9e-4017-a820-143420717b78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:19.176072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.175913    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls" pod "3739c5da-cc9e-4017-a820-143420717b78" (UID: "3739c5da-cc9e-4017-a820-143420717b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:19.183924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.183893    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj" pod "3739c5da-cc9e-4017-a820-143420717b78" (UID: "3739c5da-cc9e-4017-a820-143420717b78"). InnerVolumeSpecName "kube-api-access-5t9dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:19.269195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.269163    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:19.269195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.269191    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3739c5da-cc9e-4017-a820-143420717b78-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:19.269350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.269202    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t9dj\" (UniqueName: \"kubernetes.io/projected/3739c5da-cc9e-4017-a820-143420717b78-kube-api-access-5t9dj\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:19.269350 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.269212    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3739c5da-cc9e-4017-a820-143420717b78-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:19.695635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.695599    5099 generic.go:356] "Generic (PLEG): container finished" podID="3739c5da-cc9e-4017-a820-143420717b78" containerID="98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6" exitCode=0
Nov 02 20:33:19.696112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.695658    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"
Nov 02 20:33:19.696112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.695685    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" event={"ID":"3739c5da-cc9e-4017-a820-143420717b78","Type":"ContainerDied","Data":"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"}
Nov 02 20:33:19.696112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.695796    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng" event={"ID":"3739c5da-cc9e-4017-a820-143420717b78","Type":"ContainerDied","Data":"7b9ed2ea85edcfb53a4bb2dfcef2907096ebbf75e5f17917366414cc8efee8aa"}
Nov 02 20:33:19.696112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.695833    5099 scope.go:122] "RemoveContainer" containerID="98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"
Nov 02 20:33:19.706571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.706552    5099 scope.go:122] "RemoveContainer" containerID="98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"
Nov 02 20:33:19.706868 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:19.706838    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6\": container with ID starting with 98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6 not found: ID does not exist" containerID="98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"
Nov 02 20:33:19.706937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.706870    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6"} err="failed to get container status \"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6\": rpc error: code = NotFound desc = could not find container \"98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6\": container with ID starting with 98398b7a949d88bb213a684473fcb741116c2a72062546e99a0fa19d68fefeb6 not found: ID does not exist"
Nov 02 20:33:19.853380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.853350    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:33:19.900724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:19.900693    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a51b3f-7c9b6598b8-8dkng"]
Nov 02 20:33:21.261157 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:21.261118    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3739c5da-cc9e-4017-a820-143420717b78" path="/var/lib/kubelet/pods/3739c5da-cc9e-4017-a820-143420717b78/volumes"
Nov 02 20:33:21.576624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:21.576552    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:33:22.774347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.774308    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:33:22.775458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.775437    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3739c5da-cc9e-4017-a820-143420717b78" containerName="maas-api"
Nov 02 20:33:22.775535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.775461    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3739c5da-cc9e-4017-a820-143420717b78" containerName="maas-api"
Nov 02 20:33:22.775680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.775667    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3739c5da-cc9e-4017-a820-143420717b78" containerName="maas-api"
Nov 02 20:33:22.781268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.781246    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.870011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.869978    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:33:22.911708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.911670    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2b3b9e-openshift-default-dockercfg-l8wjk\""
Nov 02 20:33:22.922379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922341    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922382    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922458    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922501    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922525    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkp4\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922698 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922586    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922778    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922836    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:22.922953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:22.922938    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.027858    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.027920    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.027959    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028010    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028045    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkp4\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028083    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028303    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028364    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.028860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.028544    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.029475 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.029275    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.032814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.030073    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.032814 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.030335    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.046256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.045142    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.046256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.045889    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.047183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.047158    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.072468 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.065841    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.101854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.101807    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.102107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.102088    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkp4\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\") pod \"e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") " pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.399021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.398988    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:23.832006 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.830577    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:33:23.835401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.835292    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:23.835401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.835380    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:23.835570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:23.835419    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:33:24.742684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:24.742646    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerStarted","Data":"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"}
Nov 02 20:33:24.742900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:24.742748    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerStarted","Data":"eeef54257c316c4c32c4e08e486fbdbef53966422729d03c07df8577685969b0"}
Nov 02 20:33:24.924024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:24.923972    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podStartSLOduration=2.923956476 podStartE2EDuration="2.923956476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:24.921019731 +0000 UTC m=+2493.795290621" watchObservedRunningTime="2026-11-02 20:33:24.923956476 +0000 UTC m=+2493.798227361"
Nov 02 20:33:25.400832 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:25.400798    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:25.407292 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:25.407260    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:25.787186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:25.787083    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:25.803109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:25.803078    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:33:27.628091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:27.628057    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:33:27.628575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:27.628299    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" containerID="cri-o://94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e" gracePeriod=30
Nov 02 20:33:30.333125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:30.333094    5099 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:33:30.333623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:30.333151    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:33:32.929478 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.929451    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:33:32.934533 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.934507    5099 generic.go:356] "Generic (PLEG): container finished" podID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerID="94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e" exitCode=0
Nov 02 20:33:32.934657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.934557    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"
Nov 02 20:33:32.934657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.934600    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerDied","Data":"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"}
Nov 02 20:33:32.934657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.934654    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6" event={"ID":"924b5181-cabf-49bf-98aa-e6b4e0d87a1d","Type":"ContainerDied","Data":"d29710e6f63e73befb580df438a5bfa2f4754ea1c8e35b06c6d001a38fe75037"}
Nov 02 20:33:32.934774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.934678    5099 scope.go:122] "RemoveContainer" containerID="94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"
Nov 02 20:33:32.945249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.945206    5099 scope.go:122] "RemoveContainer" containerID="d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"
Nov 02 20:33:32.955284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.955265    5099 scope.go:122] "RemoveContainer" containerID="94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"
Nov 02 20:33:32.955545 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:32.955525    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e\": container with ID starting with 94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e not found: ID does not exist" containerID="94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"
Nov 02 20:33:32.955592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.955556    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e"} err="failed to get container status \"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e\": rpc error: code = NotFound desc = could not find container \"94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e\": container with ID starting with 94f3a1bcb457bbf2f98c4d7a8d2483736fe820c573fcc093787ef18f448e5d0e not found: ID does not exist"
Nov 02 20:33:32.955592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.955573    5099 scope.go:122] "RemoveContainer" containerID="d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"
Nov 02 20:33:32.955834 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:32.955817    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04\": container with ID starting with d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04 not found: ID does not exist" containerID="d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"
Nov 02 20:33:32.955891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.955841    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04"} err="failed to get container status \"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04\": rpc error: code = NotFound desc = could not find container \"d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04\": container with ID starting with d8b99f4d59763b37712ae00d9ad3b0a5c69b8a31583bf638ae3c7179be5a7e04 not found: ID does not exist"
Nov 02 20:33:32.975315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975290    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975337    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975360    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975407    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975436    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975457    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975486    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975564    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975751    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\") pod \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\" (UID: \"924b5181-cabf-49bf-98aa-e6b4e0d87a1d\") "
Nov 02 20:33:32.975921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975841    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:32.975921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975860    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:32.975921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975876    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:32.976058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975863    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:32.976058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.975963    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:32.976593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.976568    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:32.976707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.976598    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:32.976707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.976615    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:32.976707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.976630    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:32.976707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.976645    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:32.977666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.977631    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:33:32.977959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.977924    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "kube-api-access-tpx24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:32.978049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.977967    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:32.978049 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:32.978005    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy" pod "924b5181-cabf-49bf-98aa-e6b4e0d87a1d" (UID: "924b5181-cabf-49bf-98aa-e6b4e0d87a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:33.078213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.078174    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:33.078213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.078205    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:33.078213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.078215    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:33.078509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.078224    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpx24\" (UniqueName: \"kubernetes.io/projected/924b5181-cabf-49bf-98aa-e6b4e0d87a1d-kube-api-access-tpx24\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:33.086813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.086761    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:33:33.404988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.404958    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:33:33.408445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:33.408422    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a51b3f-openshift-default-667894655f-8v4h6"]
Nov 02 20:33:35.260222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:35.260184    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" path="/var/lib/kubelet/pods/924b5181-cabf-49bf-98aa-e6b4e0d87a1d/volumes"
Nov 02 20:33:40.871717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:40.871678    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:33:40.984494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:40.984439    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:33:44.656795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:44.656727    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:33:44.657317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:44.657018    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy" containerID="cri-o://42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80" gracePeriod=30
Nov 02 20:33:45.872831 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:45.872761    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing" containerID="cri-o://9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c" gracePeriod=25
Nov 02 20:33:45.985154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:45.985086    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing" containerID="cri-o://b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef" gracePeriod=25
Nov 02 20:33:46.101961 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:46.101912    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:33:47.520085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:47.520043    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:33:47.621676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:47.621640    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:33:48.113651 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.113605    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" containerID="cri-o://7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e" gracePeriod=45
Nov 02 20:33:48.170817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.170762    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": dial tcp 10.132.0.115:8000: connect: connection refused"
Nov 02 20:33:48.393176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.393147    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:33:48.524432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524380    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.524919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524521    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.524919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524565    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.524919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524632    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\" (UniqueName: \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.524919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524824    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.525213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524955    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\") pod \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\" (UID: \"9e6dc5a8-18b5-4715-8ff6-584df80cded0\") "
Nov 02 20:33:48.525213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.524963    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:48.525527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.525497    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:48.525760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.525658    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:48.525760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.525666    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:48.525760 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.525729    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:48.527827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.527752    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:48.529470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.529444    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "kube-api-access-6b82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:48.530373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.530353    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs" pod "9e6dc5a8-18b5-4715-8ff6-584df80cded0" (UID: "9e6dc5a8-18b5-4715-8ff6-584df80cded0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:33:48.630571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.630466    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:48.630571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.630511    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6dc5a8-18b5-4715-8ff6-584df80cded0-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:48.630571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.630527    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b82s\" (UniqueName: \"kubernetes.io/projected/9e6dc5a8-18b5-4715-8ff6-584df80cded0-kube-api-access-6b82s\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:48.631399 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:48.630573    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6dc5a8-18b5-4715-8ff6-584df80cded0-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:49.205808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.205556    5099 generic.go:356] "Generic (PLEG): container finished" podID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerID="7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e" exitCode=0
Nov 02 20:33:49.205808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.205713    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" event={"ID":"9e6dc5a8-18b5-4715-8ff6-584df80cded0","Type":"ContainerDied","Data":"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"}
Nov 02 20:33:49.205808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.205755    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw" event={"ID":"9e6dc5a8-18b5-4715-8ff6-584df80cded0","Type":"ContainerDied","Data":"e8ab9d164810cd6dcd413d311ce872d6d549e84f13f85e2b67a5f682eb5d7365"}
Nov 02 20:33:49.205808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.205777    5099 scope.go:122] "RemoveContainer" containerID="7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"
Nov 02 20:33:49.206188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.205979    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"
Nov 02 20:33:49.279260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.279231    5099 scope.go:122] "RemoveContainer" containerID="7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"
Nov 02 20:33:49.279598 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:49.279575    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e\": container with ID starting with 7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e not found: ID does not exist" containerID="7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"
Nov 02 20:33:49.279680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.279612    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e"} err="failed to get container status \"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e\": rpc error: code = NotFound desc = could not find container \"7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e\": container with ID starting with 7474b451aefa398f5d000b3cc4fc965d11c4fc492ce9a0294851bccd611eb06e not found: ID does not exist"
Nov 02 20:33:49.385500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.385463    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:33:49.431917 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.431882    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-2-w1-accdf5-kserve-6d5856b957-clzvw"]
Nov 02 20:33:49.966912 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:49.966887    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:33:50.056087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056056    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056140    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056251    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056282    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056369    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:50.056477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056399    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056492    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056523    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056603    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:50.056702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056638    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.056702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.056681    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\") pod \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\" (UID: \"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1\") "
Nov 02 20:33:50.057233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057011    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:50.057415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057374    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:50.057415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057392    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:50.057776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057718    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.057776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057751    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.057776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057766    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.058018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.057851    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.060050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.060028    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "kube-api-access-r48xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:50.068509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.068462    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:33:50.068635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.068599    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:50.070191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.070161    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo" pod "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" (UID: "d2e2ef40-1e0a-46a1-b942-1b0be8907ff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:33:50.158935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.158900    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48xr\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-kube-api-access-r48xr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.158935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.158928    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.158935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.158937    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.159176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.158953    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.159176 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.158965    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:50.215293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.215190    5099 generic.go:356] "Generic (PLEG): container finished" podID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerID="42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80" exitCode=0
Nov 02 20:33:50.215293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.215242    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"
Nov 02 20:33:50.215293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.215274    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerDied","Data":"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"}
Nov 02 20:33:50.215578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.215330    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2" event={"ID":"d2e2ef40-1e0a-46a1-b942-1b0be8907ff1","Type":"ContainerDied","Data":"5e167780a9eed634a9082addfe412dc02ecdb611762f6143d9c70f7098be4723"}
Nov 02 20:33:50.215578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.215353    5099 scope.go:122] "RemoveContainer" containerID="42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"
Nov 02 20:33:50.227075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.227052    5099 scope.go:122] "RemoveContainer" containerID="f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"
Nov 02 20:33:50.236589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.236569    5099 scope.go:122] "RemoveContainer" containerID="42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"
Nov 02 20:33:50.236911 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:50.236886    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80\": container with ID starting with 42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80 not found: ID does not exist" containerID="42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"
Nov 02 20:33:50.236992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.236923    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80"} err="failed to get container status \"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80\": rpc error: code = NotFound desc = could not find container \"42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80\": container with ID starting with 42cb97866565484cabfc8c17162fe7edad6475ce9cd54cbe68fdcb3919976d80 not found: ID does not exist"
Nov 02 20:33:50.236992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.236948    5099 scope.go:122] "RemoveContainer" containerID="f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"
Nov 02 20:33:50.237196 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:50.237175    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913\": container with ID starting with f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913 not found: ID does not exist" containerID="f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"
Nov 02 20:33:50.237259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.237205    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913"} err="failed to get container status \"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913\": rpc error: code = NotFound desc = could not find container \"f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913\": container with ID starting with f95afddff982bf3a6ae24d13f74b6ab03836823497a692c08dca1fa4955f2913 not found: ID does not exist"
Nov 02 20:33:50.373924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.373890    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:33:50.377346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:50.377319    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-9t2t2"]
Nov 02 20:33:51.258291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:51.258253    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" path="/var/lib/kubelet/pods/9e6dc5a8-18b5-4715-8ff6-584df80cded0/volumes"
Nov 02 20:33:51.258984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:51.258963    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" path="/var/lib/kubelet/pods/d2e2ef40-1e0a-46a1-b942-1b0be8907ff1/volumes"
Nov 02 20:33:52.008719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.008676    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:33:52.008948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.008680    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:33:52.521278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.521237    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" podUID="9b15509e-20b1-4d38-bf80-db208b0e3b90" containerName="payload-processing" containerID="cri-o://481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f" gracePeriod=25
Nov 02 20:33:52.622659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.622593    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" podUID="46e56843-6e9d-43ad-b0bf-3c114523f415" containerName="payload-pre-processing" containerID="cri-o://9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a" gracePeriod=25
Nov 02 20:33:52.820187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.820159    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:33:52.892440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.892400    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\" (UniqueName: \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\") pod \"9b15509e-20b1-4d38-bf80-db208b0e3b90\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") "
Nov 02 20:33:52.892664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.892501    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\") pod \"9b15509e-20b1-4d38-bf80-db208b0e3b90\" (UID: \"9b15509e-20b1-4d38-bf80-db208b0e3b90\") "
Nov 02 20:33:52.893734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.893691    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume" pod "9b15509e-20b1-4d38-bf80-db208b0e3b90" (UID: "9b15509e-20b1-4d38-bf80-db208b0e3b90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:52.896293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.896263    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf" pod "9b15509e-20b1-4d38-bf80-db208b0e3b90" (UID: "9b15509e-20b1-4d38-bf80-db208b0e3b90"). InnerVolumeSpecName "kube-api-access-krnkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:52.919139 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.919115    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:33:52.994342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.994310    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\") pod \"46e56843-6e9d-43ad-b0bf-3c114523f415\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") "
Nov 02 20:33:52.994519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.994363    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\" (UniqueName: \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\") pod \"46e56843-6e9d-43ad-b0bf-3c114523f415\" (UID: \"46e56843-6e9d-43ad-b0bf-3c114523f415\") "
Nov 02 20:33:52.994700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.994680    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnkf\" (UniqueName: \"kubernetes.io/projected/9b15509e-20b1-4d38-bf80-db208b0e3b90-kube-api-access-krnkf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:52.994767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.994702    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15509e-20b1-4d38-bf80-db208b0e3b90-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:52.994842 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.994793    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume" pod "46e56843-6e9d-43ad-b0bf-3c114523f415" (UID: "46e56843-6e9d-43ad-b0bf-3c114523f415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:33:52.996472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:52.996451    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt" pod "46e56843-6e9d-43ad-b0bf-3c114523f415" (UID: "46e56843-6e9d-43ad-b0bf-3c114523f415"). InnerVolumeSpecName "kube-api-access-lpmwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:33:53.096184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.096105    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e56843-6e9d-43ad-b0bf-3c114523f415-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:53.096184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.096134    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpmwt\" (UniqueName: \"kubernetes.io/projected/46e56843-6e9d-43ad-b0bf-3c114523f415-kube-api-access-lpmwt\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:33:53.240841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.240772    5099 generic.go:356] "Generic (PLEG): container finished" podID="9b15509e-20b1-4d38-bf80-db208b0e3b90" containerID="481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f" exitCode=0
Nov 02 20:33:53.241076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.240852    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"
Nov 02 20:33:53.241076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.240859    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" event={"ID":"9b15509e-20b1-4d38-bf80-db208b0e3b90","Type":"ContainerDied","Data":"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"}
Nov 02 20:33:53.241076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.240933    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd" event={"ID":"9b15509e-20b1-4d38-bf80-db208b0e3b90","Type":"ContainerDied","Data":"dc38bfb28282c4a00b25c38fa2d52ce3e881ca948cf9f71a4a42dd42dae80162"}
Nov 02 20:33:53.241076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.240955    5099 scope.go:122] "RemoveContainer" containerID="481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"
Nov 02 20:33:53.242637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.242611    5099 generic.go:356] "Generic (PLEG): container finished" podID="46e56843-6e9d-43ad-b0bf-3c114523f415" containerID="9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a" exitCode=0
Nov 02 20:33:53.242810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.242659    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"
Nov 02 20:33:53.242810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.242689    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" event={"ID":"46e56843-6e9d-43ad-b0bf-3c114523f415","Type":"ContainerDied","Data":"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"}
Nov 02 20:33:53.242810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.242726    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd" event={"ID":"46e56843-6e9d-43ad-b0bf-3c114523f415","Type":"ContainerDied","Data":"5916d3f05b952f3c2347b180272c792b897820925a79633754f4b5753d4d3e1f"}
Nov 02 20:33:53.254225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.254201    5099 scope.go:122] "RemoveContainer" containerID="481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"
Nov 02 20:33:53.254598 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:53.254574    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f\": container with ID starting with 481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f not found: ID does not exist" containerID="481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"
Nov 02 20:33:53.254674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.254605    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f"} err="failed to get container status \"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f\": rpc error: code = NotFound desc = could not find container \"481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f\": container with ID starting with 481d9b93bad45583a0532db98568a4d65d1f3f4848d2cc3afe4b1a9eaf5d9b9f not found: ID does not exist"
Nov 02 20:33:53.254674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.254624    5099 scope.go:122] "RemoveContainer" containerID="9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"
Nov 02 20:33:53.265499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.265482    5099 scope.go:122] "RemoveContainer" containerID="9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"
Nov 02 20:33:53.265765 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:33:53.265745    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a\": container with ID starting with 9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a not found: ID does not exist" containerID="9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"
Nov 02 20:33:53.265827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.265773    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a"} err="failed to get container status \"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a\": rpc error: code = NotFound desc = could not find container \"9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a\": container with ID starting with 9b6c43e3174fb978fd938ca6155031019e860cb2541eb4834eccdcc60805983a not found: ID does not exist"
Nov 02 20:33:53.396537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.396502    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:33:53.401821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.401776    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4dc8ca-59f5cc7b47-xbqgd"]
Nov 02 20:33:53.504549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.504519    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:33:53.509196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:53.509165    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4dc8ca-66c89f4685-724hd"]
Nov 02 20:33:55.260595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:55.260560    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e56843-6e9d-43ad-b0bf-3c114523f415" path="/var/lib/kubelet/pods/46e56843-6e9d-43ad-b0bf-3c114523f415/volumes"
Nov 02 20:33:55.261241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:55.261222    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b15509e-20b1-4d38-bf80-db208b0e3b90" path="/var/lib/kubelet/pods/9b15509e-20b1-4d38-bf80-db208b0e3b90/volumes"
Nov 02 20:33:56.225002 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:56.224966    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:33:56.225255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:33:56.225229    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy" containerID="cri-o://86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3" gracePeriod=30
Nov 02 20:34:01.497953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.497885    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:34:01.508453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508428    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508522    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508553    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508581    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508604    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508644    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508795    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508807    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:01.508892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508844    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.508892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.508885    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\") pod \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\" (UID: \"79109b21-b5d9-4195-9597-ae7f7e36cb8d\") "
Nov 02 20:34:01.509121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509028    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:01.509337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509314    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:01.509433 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509357    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:01.509660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509534    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.509660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509557    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.509660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509573    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.509660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509587    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.509660 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.509637    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:01.511107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.511080    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:34:01.511107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.511091    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:01.511265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.511234    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "kube-api-access-k82vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:01.511328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.511310    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy" pod "79109b21-b5d9-4195-9597-ae7f7e36cb8d" (UID: "79109b21-b5d9-4195-9597-ae7f7e36cb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:34:01.611019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.610987    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.611019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.611013    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.611019 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.611022    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.611261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.611031    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:01.611261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:01.611040    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k82vn\" (UniqueName: \"kubernetes.io/projected/79109b21-b5d9-4195-9597-ae7f7e36cb8d-kube-api-access-k82vn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:02.008716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.008677    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:34:02.008716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.008712    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:34:02.318648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.318553    5099 generic.go:356] "Generic (PLEG): container finished" podID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerID="86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3" exitCode=0
Nov 02 20:34:02.318648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.318629    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"
Nov 02 20:34:02.318893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.318639    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerDied","Data":"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"}
Nov 02 20:34:02.318893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.318713    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9" event={"ID":"79109b21-b5d9-4195-9597-ae7f7e36cb8d","Type":"ContainerDied","Data":"eb733567ba25cb6bae41dfffc24815fed07b2007a792595dce7b385a13e446b2"}
Nov 02 20:34:02.318893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.318744    5099 scope.go:122] "RemoveContainer" containerID="86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"
Nov 02 20:34:02.330205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.330178    5099 scope.go:122] "RemoveContainer" containerID="fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"
Nov 02 20:34:02.340070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.340046    5099 scope.go:122] "RemoveContainer" containerID="86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"
Nov 02 20:34:02.340390 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:02.340357    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3\": container with ID starting with 86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3 not found: ID does not exist" containerID="86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"
Nov 02 20:34:02.340512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.340399    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3"} err="failed to get container status \"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3\": rpc error: code = NotFound desc = could not find container \"86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3\": container with ID starting with 86c7167497d27e098eb8854dc5ea891eceba3bb9245e19011fae5feb63ccc1d3 not found: ID does not exist"
Nov 02 20:34:02.340512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.340423    5099 scope.go:122] "RemoveContainer" containerID="fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"
Nov 02 20:34:02.340734 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:02.340716    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20\": container with ID starting with fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20 not found: ID does not exist" containerID="fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"
Nov 02 20:34:02.340822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.340741    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20"} err="failed to get container status \"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20\": rpc error: code = NotFound desc = could not find container \"fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20\": container with ID starting with fd0262ee48f4de34235f627e10801757c6fc217c8ad1024191d0eb0b4840ac20 not found: ID does not exist"
Nov 02 20:34:02.476900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.476868    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:34:02.523476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:02.523442    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62d287-openshift-default-6f4ffc6985-pkxx9"]
Nov 02 20:34:03.260541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:03.260504    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" path="/var/lib/kubelet/pods/79109b21-b5d9-4195-9597-ae7f7e36cb8d/volumes"
Nov 02 20:34:06.180204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.180173    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:34:06.279391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.279363    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\") pod \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") "
Nov 02 20:34:06.279531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.279486    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\" (UniqueName: \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\") pod \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\" (UID: \"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46\") "
Nov 02 20:34:06.279751 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.279725    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume" pod "3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" (UID: "3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:06.280039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.280013    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:06.281708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.281680    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf" pod "3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" (UID: "3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46"). InnerVolumeSpecName "kube-api-access-xsdlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:06.289298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.289274    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:34:06.349532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.349497    5099 generic.go:356] "Generic (PLEG): container finished" podID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerID="9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c" exitCode=0
Nov 02 20:34:06.349738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.349553    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"
Nov 02 20:34:06.349839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.349746    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" event={"ID":"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46","Type":"ContainerDied","Data":"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"}
Nov 02 20:34:06.349839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.349809    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw" event={"ID":"3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46","Type":"ContainerDied","Data":"a6b8fa885cbc1658ba0a19fcc409641352b02a69fde445d298ed1c7139df18d5"}
Nov 02 20:34:06.349969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.349845    5099 scope.go:122] "RemoveContainer" containerID="9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"
Nov 02 20:34:06.351455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.351430    5099 generic.go:356] "Generic (PLEG): container finished" podID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerID="b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef" exitCode=0
Nov 02 20:34:06.351582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.351519    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" event={"ID":"8065982f-c65d-489d-ac49-017ed5f79b9d","Type":"ContainerDied","Data":"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"}
Nov 02 20:34:06.351582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.351569    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k" event={"ID":"8065982f-c65d-489d-ac49-017ed5f79b9d","Type":"ContainerDied","Data":"f31d6cec254fab65a38d6f225d532a55450e3b58fbdfec03f5b89712c6c6ce48"}
Nov 02 20:34:06.351721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.351621    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"
Nov 02 20:34:06.361225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.361201    5099 scope.go:122] "RemoveContainer" containerID="9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"
Nov 02 20:34:06.361519 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:06.361497    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c\": container with ID starting with 9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c not found: ID does not exist" containerID="9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"
Nov 02 20:34:06.361579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.361528    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c"} err="failed to get container status \"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c\": rpc error: code = NotFound desc = could not find container \"9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c\": container with ID starting with 9847703e1aa4507c641fcce4b33deb5dde929a6b9f2e7c66f793b32820b4ea1c not found: ID does not exist"
Nov 02 20:34:06.361579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.361547    5099 scope.go:122] "RemoveContainer" containerID="b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"
Nov 02 20:34:06.370956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.370933    5099 scope.go:122] "RemoveContainer" containerID="b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"
Nov 02 20:34:06.371274 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:06.371254    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef\": container with ID starting with b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef not found: ID does not exist" containerID="b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"
Nov 02 20:34:06.371335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.371284    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef"} err="failed to get container status \"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef\": rpc error: code = NotFound desc = could not find container \"b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef\": container with ID starting with b261e09df91e344fba98e21ab1578bc00d32f3d2b759cd884d2fbd1411d996ef not found: ID does not exist"
Nov 02 20:34:06.380667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.380646    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\" (UniqueName: \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\") pod \"8065982f-c65d-489d-ac49-017ed5f79b9d\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") "
Nov 02 20:34:06.380854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.380837    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\") pod \"8065982f-c65d-489d-ac49-017ed5f79b9d\" (UID: \"8065982f-c65d-489d-ac49-017ed5f79b9d\") "
Nov 02 20:34:06.381192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.381165    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume" pod "8065982f-c65d-489d-ac49-017ed5f79b9d" (UID: "8065982f-c65d-489d-ac49-017ed5f79b9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:06.381587 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.381564    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8065982f-c65d-489d-ac49-017ed5f79b9d-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:06.381679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.381592    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdlf\" (UniqueName: \"kubernetes.io/projected/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46-kube-api-access-xsdlf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:06.383029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.383004    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx" pod "8065982f-c65d-489d-ac49-017ed5f79b9d" (UID: "8065982f-c65d-489d-ac49-017ed5f79b9d"). InnerVolumeSpecName "kube-api-access-nbxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:06.483424 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.483322    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxbx\" (UniqueName: \"kubernetes.io/projected/8065982f-c65d-489d-ac49-017ed5f79b9d-kube-api-access-nbxbx\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:06.506078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.506043    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:34:06.509396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.509371    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-accdf5-76dd7c7df-tc7qw"]
Nov 02 20:34:06.770650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.770555    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:34:06.775563 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:06.775530    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-accdf5-6fd488bc79-f7j4k"]
Nov 02 20:34:07.262164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:07.262130    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" path="/var/lib/kubelet/pods/3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46/volumes"
Nov 02 20:34:07.262679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:07.262559    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" path="/var/lib/kubelet/pods/8065982f-c65d-489d-ac49-017ed5f79b9d/volumes"
Nov 02 20:34:09.855479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:09.855435    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:34:09.973629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:09.973595    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:34:12.881098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.881066    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:34:12.882347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882322    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882348    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882376    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882383    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882397    5099 cpu_manager.go:414] "containerMap: removing container" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882404    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882419    5099 cpu_manager.go:414] "containerMap: removing container" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882427    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882441    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882448    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882463    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9b15509e-20b1-4d38-bf80-db208b0e3b90" containerName="payload-processing"
Nov 02 20:34:12.882474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882471    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b15509e-20b1-4d38-bf80-db208b0e3b90" containerName="payload-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882481    5099 cpu_manager.go:414] "containerMap: removing container" podUID="46e56843-6e9d-43ad-b0bf-3c114523f415" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882489    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e56843-6e9d-43ad-b0bf-3c114523f415" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882511    5099 cpu_manager.go:414] "containerMap: removing container" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882519    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882537    5099 cpu_manager.go:414] "containerMap: removing container" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882545    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882566    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882574    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882591    5099 cpu_manager.go:414] "containerMap: removing container" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882600    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882803    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882822    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6dc5a8-18b5-4715-8ff6-584df80cded0" containerName="main"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882832    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882843    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="924b5181-cabf-49bf-98aa-e6b4e0d87a1d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882851    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e2ef40-1e0a-46a1-b942-1b0be8907ff1" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882863    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b15509e-20b1-4d38-bf80-db208b0e3b90" containerName="payload-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882875    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e56843-6e9d-43ad-b0bf-3c114523f415" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882886    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882897    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="8065982f-c65d-489d-ac49-017ed5f79b9d" containerName="payload-pre-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882907    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ccf63bf-fc64-4d4e-b4b4-6d028b76fa46" containerName="payload-processing"
Nov 02 20:34:12.883084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.882917    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="79109b21-b5d9-4195-9597-ae7f7e36cb8d" containerName="istio-proxy"
Nov 02 20:34:12.888319 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.888299    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:12.938726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.938695    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:34:12.981641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.981593    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0d9ec0-serving-cert\""
Nov 02 20:34:12.990310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.990274    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:12.990487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.990341    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6n2\" (UniqueName: \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:12.990568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.990546    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:12.990620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:12.990600    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.091649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.091582    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6n2\" (UniqueName: \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.091882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.091736    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.091882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.091770    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.091882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.091863    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.092020 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:13.091900    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-serving-cert: secret "maas-api-e2e-inf-a-0d9ec0-serving-cert" not found
Nov 02 20:34:13.092020 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:13.091989    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls podName:00a1bbbb-f935-471a-8f65-c7a4997c1d92 nodeName:}" failed. No retries permitted until 2026-11-02 20:34:13.59195501 +0000 UTC m=+2542.466225883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls") pod "maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" (UID: "00a1bbbb-f935-471a-8f65-c7a4997c1d92") : secret "maas-api-e2e-inf-a-0d9ec0-serving-cert" not found
Nov 02 20:34:13.092491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.092469    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.094485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.094459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.152349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.151775    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6n2\" (UniqueName: \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.601989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.601905    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.604407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.604376    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:13.809203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:13.809171    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:14.183248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:14.183210    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:34:14.184834 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:34:14.184800    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a1bbbb_f935_471a_8f65_c7a4997c1d92.slice/crio-12898d07d3502bc3099e92ca321084624e44e1db9ba7e820cc7bc49dbae5b7e2 WatchSource:0}: Error finding container 12898d07d3502bc3099e92ca321084624e44e1db9ba7e820cc7bc49dbae5b7e2: Status 404 returned error can't find the container with id 12898d07d3502bc3099e92ca321084624e44e1db9ba7e820cc7bc49dbae5b7e2
Nov 02 20:34:14.186387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:14.186363    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:34:14.456731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:14.456639    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" event={"ID":"00a1bbbb-f935-471a-8f65-c7a4997c1d92","Type":"ContainerStarted","Data":"12898d07d3502bc3099e92ca321084624e44e1db9ba7e820cc7bc49dbae5b7e2"}
Nov 02 20:34:14.855857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:14.855723    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" podUID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" containerName="payload-processing" containerID="cri-o://39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9" gracePeriod=25
Nov 02 20:34:14.974564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:14.974529    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" podUID="429aa04f-9423-4497-ad76-10359fb9fbdd" containerName="payload-pre-processing" containerID="cri-o://61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894" gracePeriod=25
Nov 02 20:34:15.147238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.147202    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:34:15.225312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.225272    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\") pod \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") "
Nov 02 20:34:15.225747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.225464    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\" (UniqueName: \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\") pod \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\" (UID: \"26d2072d-35a9-4a61-8229-f3d167f3e0fd\") "
Nov 02 20:34:15.225747 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.225692    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume" pod "26d2072d-35a9-4a61-8229-f3d167f3e0fd" (UID: "26d2072d-35a9-4a61-8229-f3d167f3e0fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:15.226221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.226196    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d2072d-35a9-4a61-8229-f3d167f3e0fd-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.230816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.229492    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h" pod "26d2072d-35a9-4a61-8229-f3d167f3e0fd" (UID: "26d2072d-35a9-4a61-8229-f3d167f3e0fd"). InnerVolumeSpecName "kube-api-access-7gh4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:15.312498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.312466    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:34:15.327774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.327727    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gh4h\" (UniqueName: \"kubernetes.io/projected/26d2072d-35a9-4a61-8229-f3d167f3e0fd-kube-api-access-7gh4h\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.428248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.428216    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\" (UniqueName: \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\") pod \"429aa04f-9423-4497-ad76-10359fb9fbdd\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") "
Nov 02 20:34:15.428454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.428349    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\") pod \"429aa04f-9423-4497-ad76-10359fb9fbdd\" (UID: \"429aa04f-9423-4497-ad76-10359fb9fbdd\") "
Nov 02 20:34:15.428727 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.428703    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume" pod "429aa04f-9423-4497-ad76-10359fb9fbdd" (UID: "429aa04f-9423-4497-ad76-10359fb9fbdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:15.428964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.428949    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429aa04f-9423-4497-ad76-10359fb9fbdd-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.430501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.430471    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz" pod "429aa04f-9423-4497-ad76-10359fb9fbdd" (UID: "429aa04f-9423-4497-ad76-10359fb9fbdd"). InnerVolumeSpecName "kube-api-access-rh6jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:15.471662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.471358    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" event={"ID":"00a1bbbb-f935-471a-8f65-c7a4997c1d92","Type":"ContainerStarted","Data":"cc21837725ef98728be93515b4f2e9c272870ebb3eb172c85340c94b53e45d8e"}
Nov 02 20:34:15.471911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.471719    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:15.473472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.473441    5099 generic.go:356] "Generic (PLEG): container finished" podID="429aa04f-9423-4497-ad76-10359fb9fbdd" containerID="61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894" exitCode=0
Nov 02 20:34:15.473609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.473483    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"
Nov 02 20:34:15.473609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.473499    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" event={"ID":"429aa04f-9423-4497-ad76-10359fb9fbdd","Type":"ContainerDied","Data":"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"}
Nov 02 20:34:15.473609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.473597    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq" event={"ID":"429aa04f-9423-4497-ad76-10359fb9fbdd","Type":"ContainerDied","Data":"70ac0c989dc8d1788f64d74f7abe7dda93093058047484fe3bcf6c265a606092"}
Nov 02 20:34:15.474073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.473637    5099 scope.go:122] "RemoveContainer" containerID="61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"
Nov 02 20:34:15.475861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.475832    5099 generic.go:356] "Generic (PLEG): container finished" podID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" containerID="39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9" exitCode=0
Nov 02 20:34:15.476030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.476010    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" event={"ID":"26d2072d-35a9-4a61-8229-f3d167f3e0fd","Type":"ContainerDied","Data":"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"}
Nov 02 20:34:15.476137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.476051    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz" event={"ID":"26d2072d-35a9-4a61-8229-f3d167f3e0fd","Type":"ContainerDied","Data":"ccbd92fef878adf94244b9879c683b31ee4d2935b8ed247a290fe4e26e6b911d"}
Nov 02 20:34:15.476137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.476114    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"
Nov 02 20:34:15.487075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.487049    5099 scope.go:122] "RemoveContainer" containerID="61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"
Nov 02 20:34:15.487418 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:15.487389    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894\": container with ID starting with 61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894 not found: ID does not exist" containerID="61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"
Nov 02 20:34:15.487490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.487427    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894"} err="failed to get container status \"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894\": rpc error: code = NotFound desc = could not find container \"61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894\": container with ID starting with 61958b1998d783c8e3ef47f8a3cb0b7f7cb01d8a00abdba60f238b67d0b5f894 not found: ID does not exist"
Nov 02 20:34:15.487490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.487444    5099 scope.go:122] "RemoveContainer" containerID="39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"
Nov 02 20:34:15.497598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.497572    5099 scope.go:122] "RemoveContainer" containerID="39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"
Nov 02 20:34:15.497900 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:15.497883    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9\": container with ID starting with 39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9 not found: ID does not exist" containerID="39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"
Nov 02 20:34:15.497945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.497909    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9"} err="failed to get container status \"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9\": rpc error: code = NotFound desc = could not find container \"39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9\": container with ID starting with 39259f5edcd3d27fb799f780beeab035888fc9ba5b23b60dbccb2a9499e0c7d9 not found: ID does not exist"
Nov 02 20:34:15.530719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.530684    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6jz\" (UniqueName: \"kubernetes.io/projected/429aa04f-9423-4497-ad76-10359fb9fbdd-kube-api-access-rh6jz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:15.580754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.580695    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" podStartSLOduration=2.612786326 podStartE2EDuration="3.58068175s" totalImagesPullingTime="967.895424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:15.578754331 +0000 UTC m=+2544.453025219" watchObservedRunningTime="2026-11-02 20:34:15.58068175 +0000 UTC m=+2544.454952638"
Nov 02 20:34:15.685432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.685348    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:34:15.690911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.690884    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5fc50d-756968bbc5-5kbvq"]
Nov 02 20:34:15.809975 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.809938    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:34:15.815428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:15.815400    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5fc50d-567d6f945f-lqfdz"]
Nov 02 20:34:16.095338 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:16.095256    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:34:17.261873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:17.261833    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" path="/var/lib/kubelet/pods/26d2072d-35a9-4a61-8229-f3d167f3e0fd/volumes"
Nov 02 20:34:17.262481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:17.262411    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429aa04f-9423-4497-ad76-10359fb9fbdd" path="/var/lib/kubelet/pods/429aa04f-9423-4497-ad76-10359fb9fbdd/volumes"
Nov 02 20:34:21.493577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:21.493542    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:34:26.433572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.433528    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:34:26.435496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.435463    5099 cpu_manager.go:414] "containerMap: removing container" podUID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" containerName="payload-processing"
Nov 02 20:34:26.435700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.435683    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" containerName="payload-processing"
Nov 02 20:34:26.435849 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.435834    5099 cpu_manager.go:414] "containerMap: removing container" podUID="429aa04f-9423-4497-ad76-10359fb9fbdd" containerName="payload-pre-processing"
Nov 02 20:34:26.435940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.435928    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429aa04f-9423-4497-ad76-10359fb9fbdd" containerName="payload-pre-processing"
Nov 02 20:34:26.436416 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.436395    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="429aa04f-9423-4497-ad76-10359fb9fbdd" containerName="payload-pre-processing"
Nov 02 20:34:26.436561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.436549    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d2072d-35a9-4a61-8229-f3d167f3e0fd" containerName="payload-processing"
Nov 02 20:34:26.453776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.453732    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.488274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.488242    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:34:26.530521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.530491    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7b1839-serving-cert\""
Nov 02 20:34:26.584235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.577951    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.584235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.578079    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.584235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.578121    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7lh\" (UniqueName: \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.584235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.578197    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.679277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.679242    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.679480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.679401    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.679546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.679502    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.679592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.679541    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7lh\" (UniqueName: \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.680279 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.680249    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.681924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.681898    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.682065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.682045    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.733719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.733637    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7lh\" (UniqueName: \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\") pod \"maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:26.772093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:26.772057    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:27.204152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:27.204115    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:34:27.205111 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:34:27.205082    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc88d7b_4f31_4b8a_9182_d2baf064d074.slice/crio-6cebabb8afc24e86cde3f05934cc3d8e4061482d249cc83eac5ff07fc18ef443 WatchSource:0}: Error finding container 6cebabb8afc24e86cde3f05934cc3d8e4061482d249cc83eac5ff07fc18ef443: Status 404 returned error can't find the container with id 6cebabb8afc24e86cde3f05934cc3d8e4061482d249cc83eac5ff07fc18ef443
Nov 02 20:34:27.577859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:27.577747    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" event={"ID":"1cc88d7b-4f31-4b8a-9182-d2baf064d074","Type":"ContainerStarted","Data":"6cebabb8afc24e86cde3f05934cc3d8e4061482d249cc83eac5ff07fc18ef443"}
Nov 02 20:34:28.586118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:28.586083    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" event={"ID":"1cc88d7b-4f31-4b8a-9182-d2baf064d074","Type":"ContainerStarted","Data":"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"}
Nov 02 20:34:28.586553 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:28.586237    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:28.695765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:28.695705    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" podStartSLOduration=1.854261292 podStartE2EDuration="2.695687498s" totalImagesPullingTime="841.426206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:28.694369144 +0000 UTC m=+2557.568640032" watchObservedRunningTime="2026-11-02 20:34:28.695687498 +0000 UTC m=+2557.569958385"
Nov 02 20:34:34.599417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:34.599332    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:34:46.115920 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:46.115879    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:34:53.536082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.536043    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:34:53.537691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.536295    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" podUID="43e1dd02-54db-420f-b605-9a7198bea27a" containerName="maas-api" containerID="cri-o://07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca" gracePeriod=30
Nov 02 20:34:53.850606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.850577    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:34:53.861602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.861569    5099 generic.go:356] "Generic (PLEG): container finished" podID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerID="6af8800b8c098f31670f1eb0da097daffd5df138a3088641c7fe0fba220b605e" exitCode=0
Nov 02 20:34:53.861725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.861678    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerDied","Data":"6af8800b8c098f31670f1eb0da097daffd5df138a3088641c7fe0fba220b605e"}
Nov 02 20:34:53.862198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.862176    5099 scope.go:122] "RemoveContainer" containerID="6af8800b8c098f31670f1eb0da097daffd5df138a3088641c7fe0fba220b605e"
Nov 02 20:34:53.864235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.864201    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:53.864329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.864291    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:53.864373 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.864338    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:53.867648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.867289    5099 generic.go:356] "Generic (PLEG): container finished" podID="43e1dd02-54db-420f-b605-9a7198bea27a" containerID="07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca" exitCode=0
Nov 02 20:34:53.867648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.867355    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" event={"ID":"43e1dd02-54db-420f-b605-9a7198bea27a","Type":"ContainerDied","Data":"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"}
Nov 02 20:34:53.867648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.867398    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m" event={"ID":"43e1dd02-54db-420f-b605-9a7198bea27a","Type":"ContainerDied","Data":"308641cd533e82ed5c433a14fa977ca56633b18a7a07825365cdbefca037e40f"}
Nov 02 20:34:53.867648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.867426    5099 scope.go:122] "RemoveContainer" containerID="07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"
Nov 02 20:34:53.867648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.867425    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"
Nov 02 20:34:53.883554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.883521    5099 scope.go:122] "RemoveContainer" containerID="07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"
Nov 02 20:34:53.883910 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:34:53.883882    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca\": container with ID starting with 07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca not found: ID does not exist" containerID="07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"
Nov 02 20:34:53.884022 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.883920    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca"} err="failed to get container status \"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca\": rpc error: code = NotFound desc = could not find container \"07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca\": container with ID starting with 07817dde9bb13c0832ab8a8b2df6c6bc6c404f17d15e3bb008d57417fe6434ca not found: ID does not exist"
Nov 02 20:34:53.914074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.892769    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") pod \"43e1dd02-54db-420f-b605-9a7198bea27a\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") "
Nov 02 20:34:53.914074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.893268    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\" (UniqueName: \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\") pod \"43e1dd02-54db-420f-b605-9a7198bea27a\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") "
Nov 02 20:34:53.914074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.893317    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\") pod \"43e1dd02-54db-420f-b605-9a7198bea27a\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") "
Nov 02 20:34:53.914074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.893401    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\") pod \"43e1dd02-54db-420f-b605-9a7198bea27a\" (UID: \"43e1dd02-54db-420f-b605-9a7198bea27a\") "
Nov 02 20:34:53.931839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.919189    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca" pod "43e1dd02-54db-420f-b605-9a7198bea27a" (UID: "43e1dd02-54db-420f-b605-9a7198bea27a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:34:53.931839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.920166    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e1dd02-54db-420f-b605-9a7198bea27a-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:53.953294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.953231    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n" pod "43e1dd02-54db-420f-b605-9a7198bea27a" (UID: "43e1dd02-54db-420f-b605-9a7198bea27a"). InnerVolumeSpecName "kube-api-access-tcz4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:34:53.953621 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.953584    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls" pod "43e1dd02-54db-420f-b605-9a7198bea27a" (UID: "43e1dd02-54db-420f-b605-9a7198bea27a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:34:53.959016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:53.958853    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs" pod "43e1dd02-54db-420f-b605-9a7198bea27a" (UID: "43e1dd02-54db-420f-b605-9a7198bea27a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:34:54.021913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.021886    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:54.021913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.021914    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcz4n\" (UniqueName: \"kubernetes.io/projected/43e1dd02-54db-420f-b605-9a7198bea27a-kube-api-access-tcz4n\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:54.022151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.021925    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43e1dd02-54db-420f-b605-9a7198bea27a-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:34:54.293761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.293724    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:34:54.299065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.299036    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4dc8ca-fc464dc56-tsz2m"]
Nov 02 20:34:54.878854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.878818    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerStarted","Data":"709427eada979df363223f5a71d65e70c9fa4cf1c7c695b576f3dd67bf11056a"}
Nov 02 20:34:54.880945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.880906    5099 generic.go:356] "Generic (PLEG): container finished" podID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerID="6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7" exitCode=0
Nov 02 20:34:54.881101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.880979    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerDied","Data":"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"}
Nov 02 20:34:54.881493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.881473    5099 scope.go:122] "RemoveContainer" containerID="6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"
Nov 02 20:34:54.883357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.883314    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:54.883357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.883403    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:54.883357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:54.883457    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:34:55.290208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.290110    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e1dd02-54db-420f-b605-9a7198bea27a" path="/var/lib/kubelet/pods/43e1dd02-54db-420f-b605-9a7198bea27a/volumes"
Nov 02 20:34:55.317356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.317327    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:34:55.317633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.317609    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:34:55.317738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.317678    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:34:55.634925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.634892    5099 scope.go:122] "RemoveContainer" containerID="651e2aede76dd08feb520706fb3155fb9c1af259ff24b8eb2a402f914c178915"
Nov 02 20:34:55.673152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.673124    5099 scope.go:122] "RemoveContainer" containerID="1565889a5e3e33e18563a4a79512c4a3f78773ba447285423993c3ddee67dd28"
Nov 02 20:34:55.687104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.687076    5099 scope.go:122] "RemoveContainer" containerID="d79751dd14d636a30dc7b1bcdd77db89da707102522384270493981dd0c53b26"
Nov 02 20:34:55.700625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.700603    5099 scope.go:122] "RemoveContainer" containerID="15ad3f0f9a839e2dc1af8d0d3b1437b50808cadbf3f99737252e6bd44980e522"
Nov 02 20:34:55.716720 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.713344    5099 scope.go:122] "RemoveContainer" containerID="d002fbf3ec32ef976590a8d1a7d103cb971b1a05950e1803c6c5b6ab3325b742"
Nov 02 20:34:55.921856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:55.921825    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerStarted","Data":"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"}
Nov 02 20:34:56.317762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:56.317682    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:34:56.317762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:56.317738    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:34:56.614616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:56.614584    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:34:56.615954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:56.615927    5099 patch_prober.go:29] interesting pod/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 20:34:56.616087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:56.615988    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 20:34:57.318043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:57.317962    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:34:57.318043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:57.318025    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:34:57.614947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:57.614919    5099 patch_prober.go:29] interesting pod/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 20:34:57.615124 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:57.614969    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 20:34:58.318118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:58.318082    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:34:58.318517 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:58.318134    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:34:58.592071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:58.591969    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:34:58.614491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:58.614455    5099 patch_prober.go:29] interesting pod/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 20:34:58.614657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:58.614505    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 20:34:59.319539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:59.319500    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:34:59.320199 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:59.320160    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:34:59.614807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:59.614753    5099 patch_prober.go:29] interesting pod/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 20:34:59.614987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:34:59.614833    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 20:35:00.317797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:00.317756    5099 patch_prober.go:29] interesting pod/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Nov 02 20:35:00.317964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:00.317838    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Nov 02 20:35:00.615102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:00.615069    5099 patch_prober.go:29] interesting pod/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 20:35:00.615480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:00.615124    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 20:35:01.324249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:01.324217    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:35:01.324564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:01.324534    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:35:01.325470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:01.325440    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:35:01.623537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:01.622497    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:35:01.624288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:01.624261    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:35:02.095079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:02.094989    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:35:02.095284 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:02.095248    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy" containerID="cri-o://ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66" gracePeriod=30
Nov 02 20:35:07.399034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.399010    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:35:07.508982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.508945    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509002    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509078    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509230    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:35:07.509301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509275    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509274    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:35:07.509473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509319    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509386    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509426    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509456    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509498    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\") pod \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\" (UID: \"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a\") "
Nov 02 20:35:07.509658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509430    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:35:07.509658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509519    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:35:07.509658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509562    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:07.510075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.509987    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.510075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.510009    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.510075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.510024    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.510075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.510039    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.510075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.510052    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.511929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.511902    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:35:07.512152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.512129    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:35:07.512230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.512180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "kube-api-access-xjfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:07.513115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.513093    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token" pod "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" (UID: "d19e0fd2-2a86-423a-be45-aaf6c12d4e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:07.611505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.611470    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfmq\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-kube-api-access-xjfmq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.611505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.611503    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.611505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.611513    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:07.611808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:07.611521    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:08.045614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.045579    5099 generic.go:356] "Generic (PLEG): container finished" podID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerID="ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66" exitCode=0
Nov 02 20:35:08.045851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.045644    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"
Nov 02 20:35:08.045851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.045662    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerDied","Data":"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"}
Nov 02 20:35:08.045851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.045729    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4" event={"ID":"d19e0fd2-2a86-423a-be45-aaf6c12d4e2a","Type":"ContainerDied","Data":"db6b275316faa72364e323def1ac5d4fb3c61c17dea2e51e96f558f280aa38a5"}
Nov 02 20:35:08.045851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.045759    5099 scope.go:122] "RemoveContainer" containerID="ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"
Nov 02 20:35:08.056876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.056858    5099 scope.go:122] "RemoveContainer" containerID="94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"
Nov 02 20:35:08.065978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.065958    5099 scope.go:122] "RemoveContainer" containerID="ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"
Nov 02 20:35:08.066247 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:08.066229    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66\": container with ID starting with ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66 not found: ID does not exist" containerID="ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"
Nov 02 20:35:08.066285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.066255    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66"} err="failed to get container status \"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66\": rpc error: code = NotFound desc = could not find container \"ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66\": container with ID starting with ed0eb0546d500a61f939767e8e6d4ee094d52156efdae75ffe276567398f0d66 not found: ID does not exist"
Nov 02 20:35:08.066285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.066272    5099 scope.go:122] "RemoveContainer" containerID="94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"
Nov 02 20:35:08.066543 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:08.066520    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a\": container with ID starting with 94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a not found: ID does not exist" containerID="94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"
Nov 02 20:35:08.066650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.066546    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a"} err="failed to get container status \"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a\": rpc error: code = NotFound desc = could not find container \"94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a\": container with ID starting with 94fb3df951843ad6d0aa6fbd907ebd1b0f400a609f2118963a8c528a45fdf34a not found: ID does not exist"
Nov 02 20:35:08.204232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.204199    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:35:08.209322 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.209293    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4dc8ca-openshift-default-7579c4667-qljm4"]
Nov 02 20:35:08.865451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.865416    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:35:08.865922 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:08.865636    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" podUID="d5bb0cee-f735-4885-bc4a-f116c8f24186" containerName="maas-api" containerID="cri-o://7480d60bc2e09a637b11bc47fffcbd0927c6b86105942d52f2d560f627193d77" gracePeriod=30
Nov 02 20:35:09.132628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.132538    5099 generic.go:356] "Generic (PLEG): container finished" podID="d5bb0cee-f735-4885-bc4a-f116c8f24186" containerID="7480d60bc2e09a637b11bc47fffcbd0927c6b86105942d52f2d560f627193d77" exitCode=0
Nov 02 20:35:09.133029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.133005    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" event={"ID":"d5bb0cee-f735-4885-bc4a-f116c8f24186","Type":"ContainerDied","Data":"7480d60bc2e09a637b11bc47fffcbd0927c6b86105942d52f2d560f627193d77"}
Nov 02 20:35:09.181479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.181453    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:35:09.262701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.262667    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca" pod "d5bb0cee-f735-4885-bc4a-f116c8f24186" (UID: "d5bb0cee-f735-4885-bc4a-f116c8f24186"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:09.264201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.262275    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\") pod \"d5bb0cee-f735-4885-bc4a-f116c8f24186\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") "
Nov 02 20:35:09.264453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.264427    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\" (UniqueName: \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\") pod \"d5bb0cee-f735-4885-bc4a-f116c8f24186\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") "
Nov 02 20:35:09.264646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.264628    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") pod \"d5bb0cee-f735-4885-bc4a-f116c8f24186\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") "
Nov 02 20:35:09.264763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.264747    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\") pod \"d5bb0cee-f735-4885-bc4a-f116c8f24186\" (UID: \"d5bb0cee-f735-4885-bc4a-f116c8f24186\") "
Nov 02 20:35:09.265382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.265362    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bb0cee-f735-4885-bc4a-f116c8f24186-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:09.267684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.267625    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls" pod "d5bb0cee-f735-4885-bc4a-f116c8f24186" (UID: "d5bb0cee-f735-4885-bc4a-f116c8f24186"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:35:09.267809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.267765    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs" pod "d5bb0cee-f735-4885-bc4a-f116c8f24186" (UID: "d5bb0cee-f735-4885-bc4a-f116c8f24186"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:35:09.280451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.280413    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm" pod "d5bb0cee-f735-4885-bc4a-f116c8f24186" (UID: "d5bb0cee-f735-4885-bc4a-f116c8f24186"). InnerVolumeSpecName "kube-api-access-7lcgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:09.286363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.286334    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" path="/var/lib/kubelet/pods/d19e0fd2-2a86-423a-be45-aaf6c12d4e2a/volumes"
Nov 02 20:35:09.366929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.366759    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:09.366929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.366827    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bb0cee-f735-4885-bc4a-f116c8f24186-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:09.366929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:09.366845    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcgm\" (UniqueName: \"kubernetes.io/projected/d5bb0cee-f735-4885-bc4a-f116c8f24186-kube-api-access-7lcgm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:10.140963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:10.140932    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"
Nov 02 20:35:10.141435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:10.140973    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck" event={"ID":"d5bb0cee-f735-4885-bc4a-f116c8f24186","Type":"ContainerDied","Data":"dca4b5aef0f18ce1e093f8a6d3e2e48cb5d363029198d17356e9f9ec7ebaef60"}
Nov 02 20:35:10.141435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:10.141041    5099 scope.go:122] "RemoveContainer" containerID="7480d60bc2e09a637b11bc47fffcbd0927c6b86105942d52f2d560f627193d77"
Nov 02 20:35:10.322050 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:10.322016    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:35:10.370942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:10.370905    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5fc50d-69f97df549-h72ck"]
Nov 02 20:35:11.258333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:11.258294    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bb0cee-f735-4885-bc4a-f116c8f24186" path="/var/lib/kubelet/pods/d5bb0cee-f735-4885-bc4a-f116c8f24186/volumes"
Nov 02 20:35:13.292775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:13.292735    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:35:13.409838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:13.407901    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:35:16.094308 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:16.094270    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:35:18.294005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.293965    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" podUID="78992bde-8c03-4475-a5b5-49cf6dc0a258" containerName="payload-processing" containerID="cri-o://6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304" gracePeriod=25
Nov 02 20:35:18.408687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.408633    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" podUID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" containerName="payload-pre-processing" containerID="cri-o://4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb" gracePeriod=25
Nov 02 20:35:18.591549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.591521    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:35:18.608495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.608465    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\" (UniqueName: \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\") pod \"78992bde-8c03-4475-a5b5-49cf6dc0a258\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") "
Nov 02 20:35:18.608654 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.608514    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\") pod \"78992bde-8c03-4475-a5b5-49cf6dc0a258\" (UID: \"78992bde-8c03-4475-a5b5-49cf6dc0a258\") "
Nov 02 20:35:18.608970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.608941    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume" pod "78992bde-8c03-4475-a5b5-49cf6dc0a258" (UID: "78992bde-8c03-4475-a5b5-49cf6dc0a258"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:18.611150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.611124    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x" pod "78992bde-8c03-4475-a5b5-49cf6dc0a258" (UID: "78992bde-8c03-4475-a5b5-49cf6dc0a258"). InnerVolumeSpecName "kube-api-access-xlv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:18.696961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.696927    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:35:18.709998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.709963    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlv5x\" (UniqueName: \"kubernetes.io/projected/78992bde-8c03-4475-a5b5-49cf6dc0a258-kube-api-access-xlv5x\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:18.709998 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.709995    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78992bde-8c03-4475-a5b5-49cf6dc0a258-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:18.811249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.811158    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\" (UniqueName: \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\") pod \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") "
Nov 02 20:35:18.811837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.811301    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\") pod \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\" (UID: \"ab5e35f5-c084-45e2-8f5e-4f6f119a9201\") "
Nov 02 20:35:18.812244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.812215    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume" pod "ab5e35f5-c084-45e2-8f5e-4f6f119a9201" (UID: "ab5e35f5-c084-45e2-8f5e-4f6f119a9201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:18.814044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.814021    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq" pod "ab5e35f5-c084-45e2-8f5e-4f6f119a9201" (UID: "ab5e35f5-c084-45e2-8f5e-4f6f119a9201"). InnerVolumeSpecName "kube-api-access-zwppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:18.912862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.912831    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwppq\" (UniqueName: \"kubernetes.io/projected/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-kube-api-access-zwppq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:18.912862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:18.912857    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5e35f5-c084-45e2-8f5e-4f6f119a9201-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:19.216669 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.216638    5099 generic.go:356] "Generic (PLEG): container finished" podID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" containerID="4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb" exitCode=0
Nov 02 20:35:19.216881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.216684    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"
Nov 02 20:35:19.216881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.216724    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" event={"ID":"ab5e35f5-c084-45e2-8f5e-4f6f119a9201","Type":"ContainerDied","Data":"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"}
Nov 02 20:35:19.216881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.216814    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk" event={"ID":"ab5e35f5-c084-45e2-8f5e-4f6f119a9201","Type":"ContainerDied","Data":"59b82f7ea0418ae748971191d492fe3ad2161212004e54518da5e0ae1ef8f854"}
Nov 02 20:35:19.216881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.216854    5099 scope.go:122] "RemoveContainer" containerID="4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"
Nov 02 20:35:19.218223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.218202    5099 generic.go:356] "Generic (PLEG): container finished" podID="78992bde-8c03-4475-a5b5-49cf6dc0a258" containerID="6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304" exitCode=0
Nov 02 20:35:19.218335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.218246    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"
Nov 02 20:35:19.218335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.218274    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" event={"ID":"78992bde-8c03-4475-a5b5-49cf6dc0a258","Type":"ContainerDied","Data":"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"}
Nov 02 20:35:19.218335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.218314    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk" event={"ID":"78992bde-8c03-4475-a5b5-49cf6dc0a258","Type":"ContainerDied","Data":"4f043f0e2be33e21960cd8e7dba89f054290df3e90730357c9cf31df7a2c8ef4"}
Nov 02 20:35:19.229390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.229370    5099 scope.go:122] "RemoveContainer" containerID="4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"
Nov 02 20:35:19.229678 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:19.229657    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb\": container with ID starting with 4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb not found: ID does not exist" containerID="4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"
Nov 02 20:35:19.229758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.229687    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb"} err="failed to get container status \"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb\": rpc error: code = NotFound desc = could not find container \"4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb\": container with ID starting with 4f7e0fb758499b4f4f27efb0b90c1d6c7e2c2d61f1a676c3cc45002f0c484fbb not found: ID does not exist"
Nov 02 20:35:19.229758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.229704    5099 scope.go:122] "RemoveContainer" containerID="6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"
Nov 02 20:35:19.241630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.241610    5099 scope.go:122] "RemoveContainer" containerID="6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"
Nov 02 20:35:19.241961 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:19.241933    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304\": container with ID starting with 6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304 not found: ID does not exist" containerID="6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"
Nov 02 20:35:19.242024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.241962    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304"} err="failed to get container status \"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304\": rpc error: code = NotFound desc = could not find container \"6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304\": container with ID starting with 6955f76e73fedb62d6c395515b741b433e18ff7c2f2871823ef5ea4e5a7ac304 not found: ID does not exist"
Nov 02 20:35:19.372538 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.372501    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:35:19.419301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.419267    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5efe19-8fb7c9975-cxjwk"]
Nov 02 20:35:19.522493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.522412    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:35:19.525543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:19.525514    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5efe19-5cf94c87c5-hg6gk"]
Nov 02 20:35:21.259981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:21.259944    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78992bde-8c03-4475-a5b5-49cf6dc0a258" path="/var/lib/kubelet/pods/78992bde-8c03-4475-a5b5-49cf6dc0a258/volumes"
Nov 02 20:35:21.260603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:21.260389    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" path="/var/lib/kubelet/pods/ab5e35f5-c084-45e2-8f5e-4f6f119a9201/volumes"
Nov 02 20:35:29.672017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:29.671971    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:35:29.783524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:29.783483    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:35:34.348499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.347893    5099 generic.go:356] "Generic (PLEG): container finished" podID="02ccbace-e464-4899-93c5-15f94b546071" containerID="843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9" exitCode=0
Nov 02 20:35:34.348499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.348155    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerDied","Data":"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"}
Nov 02 20:35:34.349836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.348663    5099 scope.go:122] "RemoveContainer" containerID="843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"
Nov 02 20:35:34.350880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.350840    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:35:34.350972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.350940    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:35:34.351038 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.350985    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:35:34.672337 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.672286    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" podUID="e8331cf1-415f-4198-b038-77b035431d10" containerName="payload-processing" containerID="cri-o://e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df" gracePeriod=25
Nov 02 20:35:34.784628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:34.784591    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" podUID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" containerName="payload-pre-processing" containerID="cri-o://1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f" gracePeriod=25
Nov 02 20:35:35.001806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.001501    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:35:35.105171 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.105141    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:35:35.185944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.185476    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\" (UniqueName: \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\") pod \"e8331cf1-415f-4198-b038-77b035431d10\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") "
Nov 02 20:35:35.185944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.185682    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\") pod \"e8331cf1-415f-4198-b038-77b035431d10\" (UID: \"e8331cf1-415f-4198-b038-77b035431d10\") "
Nov 02 20:35:35.200807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.195693    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume" pod "e8331cf1-415f-4198-b038-77b035431d10" (UID: "e8331cf1-415f-4198-b038-77b035431d10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:35.206039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.205964    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2" pod "e8331cf1-415f-4198-b038-77b035431d10" (UID: "e8331cf1-415f-4198-b038-77b035431d10"). InnerVolumeSpecName "kube-api-access-f4xz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:35.295954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.295920    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\") pod \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") "
Nov 02 20:35:35.296130 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.296020    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\" (UniqueName: \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\") pod \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\" (UID: \"46c41a76-cfc6-4b45-b9f7-570a1d2634db\") "
Nov 02 20:35:35.296683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.296658    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8331cf1-415f-4198-b038-77b035431d10-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:35.296812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.296687    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xz2\" (UniqueName: \"kubernetes.io/projected/e8331cf1-415f-4198-b038-77b035431d10-kube-api-access-f4xz2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:35.297301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.297267    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume" pod "46c41a76-cfc6-4b45-b9f7-570a1d2634db" (UID: "46c41a76-cfc6-4b45-b9f7-570a1d2634db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:35:35.299923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.299891    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf" pod "46c41a76-cfc6-4b45-b9f7-570a1d2634db" (UID: "46c41a76-cfc6-4b45-b9f7-570a1d2634db"). InnerVolumeSpecName "kube-api-access-dlfqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:35:35.375138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.375101    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerStarted","Data":"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"}
Nov 02 20:35:35.376674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.376639    5099 generic.go:356] "Generic (PLEG): container finished" podID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" containerID="1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f" exitCode=0
Nov 02 20:35:35.376840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.376686    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"
Nov 02 20:35:35.376840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.376740    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" event={"ID":"46c41a76-cfc6-4b45-b9f7-570a1d2634db","Type":"ContainerDied","Data":"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"}
Nov 02 20:35:35.376959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.376840    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q" event={"ID":"46c41a76-cfc6-4b45-b9f7-570a1d2634db","Type":"ContainerDied","Data":"b2300ac55358756b5b52f07f1679d2907ddd724efb4c85df23da35b60fd74b65"}
Nov 02 20:35:35.376959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.376877    5099 scope.go:122] "RemoveContainer" containerID="1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"
Nov 02 20:35:35.384689 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.384663    5099 generic.go:356] "Generic (PLEG): container finished" podID="e8331cf1-415f-4198-b038-77b035431d10" containerID="e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df" exitCode=0
Nov 02 20:35:35.384801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.384752    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" event={"ID":"e8331cf1-415f-4198-b038-77b035431d10","Type":"ContainerDied","Data":"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"}
Nov 02 20:35:35.384801 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.384773    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"
Nov 02 20:35:35.384887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.384837    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8" event={"ID":"e8331cf1-415f-4198-b038-77b035431d10","Type":"ContainerDied","Data":"94ca53d9bf63afc0c218e7e1ff0df85d4b503cf50ef9208942600d6357f9ae29"}
Nov 02 20:35:35.398270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.398246    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfqf\" (UniqueName: \"kubernetes.io/projected/46c41a76-cfc6-4b45-b9f7-570a1d2634db-kube-api-access-dlfqf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:35.398581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.398564    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c41a76-cfc6-4b45-b9f7-570a1d2634db-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:35:35.404181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.399848    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:35:35.404181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.399976    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:35.404181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.400026    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:35.404181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.402997    5099 scope.go:122] "RemoveContainer" containerID="1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"
Nov 02 20:35:35.404871 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:35.404844    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f\": container with ID starting with 1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f not found: ID does not exist" containerID="1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"
Nov 02 20:35:35.404953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.404889    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f"} err="failed to get container status \"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f\": rpc error: code = NotFound desc = could not find container \"1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f\": container with ID starting with 1b14a88daee05f41b395c89dfd3f757e0155bd302ffc893bdd3a005f1600417f not found: ID does not exist"
Nov 02 20:35:35.404953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.404912    5099 scope.go:122] "RemoveContainer" containerID="e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"
Nov 02 20:35:35.416015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.415998    5099 scope.go:122] "RemoveContainer" containerID="e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"
Nov 02 20:35:35.416381 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:35.416347    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df\": container with ID starting with e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df not found: ID does not exist" containerID="e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"
Nov 02 20:35:35.416491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.416390    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df"} err="failed to get container status \"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df\": rpc error: code = NotFound desc = could not find container \"e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df\": container with ID starting with e9e4b1b3729eb5216ffa4b6d1a88ce5372cd616211154c8e86bf352caeaff4df not found: ID does not exist"
Nov 02 20:35:35.591191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.591098    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:35:35.594024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.593998    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-40425f-6fd87d77d5-zjlh8"]
Nov 02 20:35:35.703313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.703274    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:35:35.708649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:35.708621    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-40425f-867f677df9-bn48q"]
Nov 02 20:35:36.400219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:36.400185    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:36.400724 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:36.400258    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:37.276256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:37.276225    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" path="/var/lib/kubelet/pods/46c41a76-cfc6-4b45-b9f7-570a1d2634db/volumes"
Nov 02 20:35:37.276681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:37.276667    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8331cf1-415f-4198-b038-77b035431d10" path="/var/lib/kubelet/pods/e8331cf1-415f-4198-b038-77b035431d10/volumes"
Nov 02 20:35:37.399336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:37.399298    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:37.399508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:37.399356    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:38.400465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.400422    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:38.400999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.400491    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:38.874596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.874560    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:35:38.875237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875220    5099 cpu_manager.go:414] "containerMap: removing container" podUID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" containerName="payload-pre-processing"
Nov 02 20:35:38.875237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875237    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" containerName="payload-pre-processing"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875254    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875260    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875274    5099 cpu_manager.go:414] "containerMap: removing container" podUID="43e1dd02-54db-420f-b605-9a7198bea27a" containerName="maas-api"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875280    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e1dd02-54db-420f-b605-9a7198bea27a" containerName="maas-api"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875289    5099 cpu_manager.go:414] "containerMap: removing container" podUID="78992bde-8c03-4475-a5b5-49cf6dc0a258" containerName="payload-processing"
Nov 02 20:35:38.875352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875297    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78992bde-8c03-4475-a5b5-49cf6dc0a258" containerName="payload-processing"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875496    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e8331cf1-415f-4198-b038-77b035431d10" containerName="payload-processing"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875505    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8331cf1-415f-4198-b038-77b035431d10" containerName="payload-processing"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875519    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" containerName="payload-pre-processing"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875523    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" containerName="payload-pre-processing"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875537    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d5bb0cee-f735-4885-bc4a-f116c8f24186" containerName="maas-api"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875541    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bb0cee-f735-4885-bc4a-f116c8f24186" containerName="maas-api"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875553    5099 cpu_manager.go:414] "containerMap: removing container" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.875568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875557    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875698    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c41a76-cfc6-4b45-b9f7-570a1d2634db" containerName="payload-pre-processing"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875712    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875719    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e1dd02-54db-420f-b605-9a7198bea27a" containerName="maas-api"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875725    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="78992bde-8c03-4475-a5b5-49cf6dc0a258" containerName="payload-processing"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875734    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8331cf1-415f-4198-b038-77b035431d10" containerName="payload-processing"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875743    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5e35f5-c084-45e2-8f5e-4f6f119a9201" containerName="payload-pre-processing"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875750    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bb0cee-f735-4885-bc4a-f116c8f24186" containerName="maas-api"
Nov 02 20:35:38.875839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.875756    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19e0fd2-2a86-423a-be45-aaf6c12d4e2a" containerName="istio-proxy"
Nov 02 20:35:38.878984 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.878967    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:38.930821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.930777    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:35:38.969165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.958653    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv96q\" (UniqueName: \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:38.981804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.969595    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:38.989803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.977132    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-aaa3e7-dockercfg-bnpn6\""
Nov 02 20:35:38.990043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:38.977203    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-aaa3e7\""
Nov 02 20:35:39.101591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.101551    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv96q\" (UniqueName: \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:39.101927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.101902    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:39.102977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.102950    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:39.167875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.167693    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv96q\" (UniqueName: \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\") pod \"payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:39.191860 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.191826    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:39.400057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.400012    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:39.400286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.400079    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:39.584267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:39.584236    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:35:39.586606 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:35:39.586572    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a35586f_e776_4b67_b64a_d9210b875cd8.slice/crio-0712cf435e7feea6798f92a0564f82792bda272c6e576d8dc2f86bb8097c4fd9 WatchSource:0}: Error finding container 0712cf435e7feea6798f92a0564f82792bda272c6e576d8dc2f86bb8097c4fd9: Status 404 returned error can't find the container with id 0712cf435e7feea6798f92a0564f82792bda272c6e576d8dc2f86bb8097c4fd9
Nov 02 20:35:40.400328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.400289    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 20:35:40.400537 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.400343    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 20:35:40.434736 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.434697    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" event={"ID":"1a35586f-e776-4b67-b64a-d9210b875cd8","Type":"ContainerStarted","Data":"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"}
Nov 02 20:35:40.434968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.434765    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" event={"ID":"1a35586f-e776-4b67-b64a-d9210b875cd8","Type":"ContainerStarted","Data":"0712cf435e7feea6798f92a0564f82792bda272c6e576d8dc2f86bb8097c4fd9"}
Nov 02 20:35:40.434968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.434940    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:35:40.546989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.546938    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" podStartSLOduration=2.546925173 podStartE2EDuration="2.546925173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:40.544762189 +0000 UTC m=+2629.419033078" watchObservedRunningTime="2026-11-02 20:35:40.546925173 +0000 UTC m=+2629.421196061"
Nov 02 20:35:40.787479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:40.787382    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:35:41.405426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:41.405394    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:35:41.442970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:41.442940    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:35:46.094091 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:35:46.094055    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:35:51.445355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:35:51.445324    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:36:02.343803 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:02.343702    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:36:02.344260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:02.344001    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy" containerID="cri-o://4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908" gracePeriod=30
Nov 02 20:36:07.613521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.613494    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:36:07.620881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.620796    5099 generic.go:356] "Generic (PLEG): container finished" podID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerID="4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908" exitCode=0
Nov 02 20:36:07.620881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.620837    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"
Nov 02 20:36:07.621073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.620833    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerDied","Data":"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"}
Nov 02 20:36:07.621073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.620968    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9" event={"ID":"48b29987-1040-4f1b-aa93-715832dfd0a6","Type":"ContainerDied","Data":"336a0aee444ea7d4d5c0fa14416c7ddbccfd6c8167c837fe9c5098c470f2e896"}
Nov 02 20:36:07.621073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.620997    5099 scope.go:122] "RemoveContainer" containerID="4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"
Nov 02 20:36:07.630948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.630926    5099 scope.go:122] "RemoveContainer" containerID="3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"
Nov 02 20:36:07.639988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.639965    5099 scope.go:122] "RemoveContainer" containerID="4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"
Nov 02 20:36:07.640272 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:07.640252    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908\": container with ID starting with 4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908 not found: ID does not exist" containerID="4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"
Nov 02 20:36:07.640339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.640280    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908"} err="failed to get container status \"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908\": rpc error: code = NotFound desc = could not find container \"4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908\": container with ID starting with 4eca7be9da041a64d30f23ba14890d0b2208193d3ddbf4d3ea394ae2a7587908 not found: ID does not exist"
Nov 02 20:36:07.640339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.640298    5099 scope.go:122] "RemoveContainer" containerID="3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"
Nov 02 20:36:07.640573 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:07.640548    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00\": container with ID starting with 3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00 not found: ID does not exist" containerID="3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"
Nov 02 20:36:07.640635 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.640584    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00"} err="failed to get container status \"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00\": rpc error: code = NotFound desc = could not find container \"3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00\": container with ID starting with 3e6d4196693f35e35c4a6764ac915efef721e399d3b0073b494ea8b6388e3b00 not found: ID does not exist"
Nov 02 20:36:07.702603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702572    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702615    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702695    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702753    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702811    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702888    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.702972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702928    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.703150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.702996    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.703463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703038    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:07.703591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703067    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:07.703705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703111    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:07.703705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703191    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:07.703705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703549    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\") pod \"48b29987-1040-4f1b-aa93-715832dfd0a6\" (UID: \"48b29987-1040-4f1b-aa93-715832dfd0a6\") "
Nov 02 20:36:07.703953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.703652    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:07.704224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.704204    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.704310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.704228    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b29987-1040-4f1b-aa93-715832dfd0a6-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.704310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.704243    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.704310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.704259    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.704310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.704275    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.705196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.705169    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:36:07.705360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.705341    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:07.705431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.705359    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "kube-api-access-2c9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:07.705476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.705438    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token" pod "48b29987-1040-4f1b-aa93-715832dfd0a6" (UID: "48b29987-1040-4f1b-aa93-715832dfd0a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:07.805508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.805474    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.805508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.805509    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.805723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.805524    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b29987-1040-4f1b-aa93-715832dfd0a6-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:07.805723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:07.805539    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9fq\" (UniqueName: \"kubernetes.io/projected/48b29987-1040-4f1b-aa93-715832dfd0a6-kube-api-access-2c9fq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:08.036961 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:08.036919    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:36:08.044321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:08.044278    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-fghm9"]
Nov 02 20:36:09.262312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:09.262271    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" path="/var/lib/kubelet/pods/48b29987-1040-4f1b-aa93-715832dfd0a6/volumes"
Nov 02 20:36:09.733876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:09.733842    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:36:09.734062 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:09.734042    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" podUID="c205fbaa-0c0a-417a-962c-98faaff9d146" containerName="maas-api" containerID="cri-o://de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714" gracePeriod=30
Nov 02 20:36:10.064142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.064106    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:36:10.238347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.238310    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\") pod \"c205fbaa-0c0a-417a-962c-98faaff9d146\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") "
Nov 02 20:36:10.238642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.238626    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\") pod \"c205fbaa-0c0a-417a-962c-98faaff9d146\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") "
Nov 02 20:36:10.238937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.238920    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") pod \"c205fbaa-0c0a-417a-962c-98faaff9d146\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") "
Nov 02 20:36:10.239055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.239042    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\" (UniqueName: \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\") pod \"c205fbaa-0c0a-417a-962c-98faaff9d146\" (UID: \"c205fbaa-0c0a-417a-962c-98faaff9d146\") "
Nov 02 20:36:10.239603 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.239069    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca" pod "c205fbaa-0c0a-417a-962c-98faaff9d146" (UID: "c205fbaa-0c0a-417a-962c-98faaff9d146"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:10.242446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.242413    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj" pod "c205fbaa-0c0a-417a-962c-98faaff9d146" (UID: "c205fbaa-0c0a-417a-962c-98faaff9d146"). InnerVolumeSpecName "kube-api-access-7qnqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:10.250244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.250197    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs" pod "c205fbaa-0c0a-417a-962c-98faaff9d146" (UID: "c205fbaa-0c0a-417a-962c-98faaff9d146"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:10.251261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.251235    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls" pod "c205fbaa-0c0a-417a-962c-98faaff9d146" (UID: "c205fbaa-0c0a-417a-962c-98faaff9d146"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:10.341165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.341074    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:10.341712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.341678    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qnqj\" (UniqueName: \"kubernetes.io/projected/c205fbaa-0c0a-417a-962c-98faaff9d146-kube-api-access-7qnqj\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:10.341712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.341711    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c205fbaa-0c0a-417a-962c-98faaff9d146-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:10.341955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.341726    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c205fbaa-0c0a-417a-962c-98faaff9d146-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:10.672810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.672733    5099 generic.go:356] "Generic (PLEG): container finished" podID="c205fbaa-0c0a-417a-962c-98faaff9d146" containerID="de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714" exitCode=0
Nov 02 20:36:10.673036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.673018    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" event={"ID":"c205fbaa-0c0a-417a-962c-98faaff9d146","Type":"ContainerDied","Data":"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"}
Nov 02 20:36:10.673108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.673076    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr" event={"ID":"c205fbaa-0c0a-417a-962c-98faaff9d146","Type":"ContainerDied","Data":"f4ae6745b02c3356eb071f0bacf48e238ce613f8b31dd158eff00cf7792d3e62"}
Nov 02 20:36:10.673173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.673108    5099 scope.go:122] "RemoveContainer" containerID="de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"
Nov 02 20:36:10.673308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.673291    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"
Nov 02 20:36:10.713850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.713823    5099 scope.go:122] "RemoveContainer" containerID="de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"
Nov 02 20:36:10.714183 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:10.714153    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714\": container with ID starting with de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714 not found: ID does not exist" containerID="de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"
Nov 02 20:36:10.714289 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.714190    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714"} err="failed to get container status \"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714\": rpc error: code = NotFound desc = could not find container \"de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714\": container with ID starting with de7105020ad6e1ab705b2271e7befac19ef858e63d2feafb49bba43f118c7714 not found: ID does not exist"
Nov 02 20:36:10.845990 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.845953    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:36:10.891484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:10.891451    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5efe19-7b7bc568c-jvgnr"]
Nov 02 20:36:11.259991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:11.259950    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c205fbaa-0c0a-417a-962c-98faaff9d146" path="/var/lib/kubelet/pods/c205fbaa-0c0a-417a-962c-98faaff9d146/volumes"
Nov 02 20:36:16.094100 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:16.094049    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:36:18.255219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:18.255187    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:36:18.255607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:18.255383    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy" containerID="cri-o://9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478" gracePeriod=30
Nov 02 20:36:23.568313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.568283    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:36:23.666150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666110    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666161    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666194    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666221    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666353 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666311    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666442    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666494    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666541    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666590    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\") pod \"982ee009-cc01-4529-bd61-abbefa04fec6\" (UID: \"982ee009-cc01-4529-bd61-abbefa04fec6\") "
Nov 02 20:36:23.666754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666537    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:23.666754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666728    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:23.666981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.666959    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:23.667105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667064    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:23.667231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667129    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:23.667301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667272    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.667301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667291    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982ee009-cc01-4529-bd61-abbefa04fec6-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.667381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667303    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.667381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667312    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.667381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.667321    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.668950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.668926    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:23.668950 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.668940    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:23.669122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.669102    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:36:23.669178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.669132    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm" pod "982ee009-cc01-4529-bd61-abbefa04fec6" (UID: "982ee009-cc01-4529-bd61-abbefa04fec6"). InnerVolumeSpecName "kube-api-access-qhvsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:23.696759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.696724    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:36:23.703454 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.703430    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:36:23.768188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.768155    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982ee009-cc01-4529-bd61-abbefa04fec6-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.768188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.768182    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.768188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.768194    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvsm\" (UniqueName: \"kubernetes.io/projected/982ee009-cc01-4529-bd61-abbefa04fec6-kube-api-access-qhvsm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.768510 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.768206    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982ee009-cc01-4529-bd61-abbefa04fec6-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:23.774132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.774093    5099 generic.go:356] "Generic (PLEG): container finished" podID="982ee009-cc01-4529-bd61-abbefa04fec6" containerID="9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478" exitCode=0
Nov 02 20:36:23.774299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.774149    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"
Nov 02 20:36:23.774299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.774183    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerDied","Data":"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"}
Nov 02 20:36:23.774299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.774252    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4" event={"ID":"982ee009-cc01-4529-bd61-abbefa04fec6","Type":"ContainerDied","Data":"1009c8daa5084ea33887581811c4f3d4351acc47a0850891b8775c60af260a5c"}
Nov 02 20:36:23.774299 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.774275    5099 scope.go:122] "RemoveContainer" containerID="9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"
Nov 02 20:36:23.793384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.793359    5099 scope.go:122] "RemoveContainer" containerID="7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"
Nov 02 20:36:23.805809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.805767    5099 scope.go:122] "RemoveContainer" containerID="9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"
Nov 02 20:36:23.806174 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:23.806145    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478\": container with ID starting with 9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478 not found: ID does not exist" containerID="9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"
Nov 02 20:36:23.806248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.806187    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478"} err="failed to get container status \"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478\": rpc error: code = NotFound desc = could not find container \"9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478\": container with ID starting with 9a0ae35997ca21e4d7b838651a37de405b1a4e177b50aece42da9315ae138478 not found: ID does not exist"
Nov 02 20:36:23.806248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.806208    5099 scope.go:122] "RemoveContainer" containerID="7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"
Nov 02 20:36:23.806495 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:23.806473    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea\": container with ID starting with 7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea not found: ID does not exist" containerID="7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"
Nov 02 20:36:23.806555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:23.806507    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea"} err="failed to get container status \"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea\": rpc error: code = NotFound desc = could not find container \"7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea\": container with ID starting with 7e71cc3848302a703fb8eada72229f3abbed068514e137a3848945e18eb91aea not found: ID does not exist"
Nov 02 20:36:24.066514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:24.066414    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:36:24.069200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:24.069175    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5efe19-openshift-default-7b68dc759c-fwwf4"]
Nov 02 20:36:25.258490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.258458    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" path="/var/lib/kubelet/pods/982ee009-cc01-4529-bd61-abbefa04fec6/volumes"
Nov 02 20:36:25.427087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.427052    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:36:25.427270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.427246    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" podUID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" containerName="maas-api" containerID="cri-o://94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d" gracePeriod=30
Nov 02 20:36:25.729094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.729067    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:36:25.789870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.789836    5099 generic.go:356] "Generic (PLEG): container finished" podID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" containerID="94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d" exitCode=0
Nov 02 20:36:25.790067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.789888    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"
Nov 02 20:36:25.790067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.789920    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" event={"ID":"bc542c7b-c42e-4e5d-bb02-b32e78c49115","Type":"ContainerDied","Data":"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"}
Nov 02 20:36:25.790067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.789979    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj" event={"ID":"bc542c7b-c42e-4e5d-bb02-b32e78c49115","Type":"ContainerDied","Data":"aa15e3f29bdb124a00b4462f3f1aaa43bd29bd0cb04707bbe1215294469912b6"}
Nov 02 20:36:25.790067 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.790003    5099 scope.go:122] "RemoveContainer" containerID="94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"
Nov 02 20:36:25.800484 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.800466    5099 scope.go:122] "RemoveContainer" containerID="94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"
Nov 02 20:36:25.800811 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:25.800757    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d\": container with ID starting with 94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d not found: ID does not exist" containerID="94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"
Nov 02 20:36:25.800925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.800821    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d"} err="failed to get container status \"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d\": rpc error: code = NotFound desc = could not find container \"94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d\": container with ID starting with 94d779b4f34b08d7db298dfb062cee9e586c06ae6a65a24b8a64d52786e9ff4d not found: ID does not exist"
Nov 02 20:36:25.894898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.894867    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") pod \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") "
Nov 02 20:36:25.895074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.895016    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\") pod \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") "
Nov 02 20:36:25.895074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.895057    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\" (UniqueName: \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\") pod \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") "
Nov 02 20:36:25.895152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.895086    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\") pod \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\" (UID: \"bc542c7b-c42e-4e5d-bb02-b32e78c49115\") "
Nov 02 20:36:25.895500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.895467    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca" pod "bc542c7b-c42e-4e5d-bb02-b32e78c49115" (UID: "bc542c7b-c42e-4e5d-bb02-b32e78c49115"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:25.895655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.895626    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc542c7b-c42e-4e5d-bb02-b32e78c49115-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:25.897432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.897399    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls" pod "bc542c7b-c42e-4e5d-bb02-b32e78c49115" (UID: "bc542c7b-c42e-4e5d-bb02-b32e78c49115"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:25.897581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.897482    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56" pod "bc542c7b-c42e-4e5d-bb02-b32e78c49115" (UID: "bc542c7b-c42e-4e5d-bb02-b32e78c49115"). InnerVolumeSpecName "kube-api-access-prn56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:25.897681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.897663    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs" pod "bc542c7b-c42e-4e5d-bb02-b32e78c49115" (UID: "bc542c7b-c42e-4e5d-bb02-b32e78c49115"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:25.996618 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.996590    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prn56\" (UniqueName: \"kubernetes.io/projected/bc542c7b-c42e-4e5d-bb02-b32e78c49115-kube-api-access-prn56\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:25.996791 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.996616    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:25.996791 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:25.996658    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc542c7b-c42e-4e5d-bb02-b32e78c49115-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:26.219509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:26.219476    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:36:26.222457 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:26.222432    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-40425f-64bf88775d-9cpcj"]
Nov 02 20:36:27.259073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:27.259039    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" path="/var/lib/kubelet/pods/bc542c7b-c42e-4e5d-bb02-b32e78c49115/volumes"
Nov 02 20:36:30.707187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:30.707149    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:36:30.818650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:30.817375    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:36:31.774521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.774479    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:36:31.775390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775368    5099 cpu_manager.go:414] "containerMap: removing container" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775393    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775412    5099 cpu_manager.go:414] "containerMap: removing container" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775419    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775429    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c205fbaa-0c0a-417a-962c-98faaff9d146" containerName="maas-api"
Nov 02 20:36:31.775467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775438    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c205fbaa-0c0a-417a-962c-98faaff9d146" containerName="maas-api"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775480    5099 cpu_manager.go:414] "containerMap: removing container" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775487    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775496    5099 cpu_manager.go:414] "containerMap: removing container" podUID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" containerName="maas-api"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775502    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" containerName="maas-api"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775528    5099 cpu_manager.go:414] "containerMap: removing container" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775535    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775671    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775682    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc542c7b-c42e-4e5d-bb02-b32e78c49115" containerName="maas-api"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775698    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b29987-1040-4f1b-aa93-715832dfd0a6" containerName="istio-proxy"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775712    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775721    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="982ee009-cc01-4529-bd61-abbefa04fec6" containerName="istio-proxy"
Nov 02 20:36:31.775843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.775731    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c205fbaa-0c0a-417a-962c-98faaff9d146" containerName="maas-api"
Nov 02 20:36:31.781331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.781309    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.829725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.829691    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:36:31.830735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.830706    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aaa3e7-serving-cert\""
Nov 02 20:36:31.891638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.891597    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.891856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.891730    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.891856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.891833    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8c2\" (UniqueName: \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.892026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.892003    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.993014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.992969    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.993226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.993034    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.993226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.993121    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.993226 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:31.993153    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-serving-cert: secret "maas-api-e2e-inf-b-aaa3e7-serving-cert" not found
Nov 02 20:36:31.993226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.993184    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8c2\" (UniqueName: \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.993466 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:31.993246    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls podName:cd7e92a3-3591-4d98-ba7f-73215c040874 nodeName:}" failed. No retries permitted until 2026-11-02 20:36:32.493223352 +0000 UTC m=+2681.367494233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls") pod "maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" (UID: "cd7e92a3-3591-4d98-ba7f-73215c040874") : secret "maas-api-e2e-inf-b-aaa3e7-serving-cert" not found
Nov 02 20:36:31.993909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.993879    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:31.995955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:31.995929    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:32.047675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:32.047596    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8c2\" (UniqueName: \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:32.520804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:32.511398    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:32.535544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:32.535508    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") pod \"maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:32.698259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:32.698226    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:33.073008 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:36:33.072975    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7e92a3_3591_4d98_ba7f_73215c040874.slice/crio-b2b1d2f8e4787c196923e85eff650b409882294a386dd3ceef81900ec2eb27ff WatchSource:0}: Error finding container b2b1d2f8e4787c196923e85eff650b409882294a386dd3ceef81900ec2eb27ff: Status 404 returned error can't find the container with id b2b1d2f8e4787c196923e85eff650b409882294a386dd3ceef81900ec2eb27ff
Nov 02 20:36:33.073543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:33.073473    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:36:33.848722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:33.848682    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" event={"ID":"cd7e92a3-3591-4d98-ba7f-73215c040874","Type":"ContainerStarted","Data":"b2b1d2f8e4787c196923e85eff650b409882294a386dd3ceef81900ec2eb27ff"}
Nov 02 20:36:34.106341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:34.106238    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:36:34.106850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:34.106455    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy" containerID="cri-o://e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00" gracePeriod=30
Nov 02 20:36:34.859434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:34.859401    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" event={"ID":"cd7e92a3-3591-4d98-ba7f-73215c040874","Type":"ContainerStarted","Data":"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"}
Nov 02 20:36:34.859633 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:34.859494    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:34.976156 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:34.976090    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" podStartSLOduration=3.136504134 podStartE2EDuration="3.976071602s" totalImagesPullingTime="839.567468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:34.971361641 +0000 UTC m=+2683.845632567" watchObservedRunningTime="2026-11-02 20:36:34.976071602 +0000 UTC m=+2683.850342494"
Nov 02 20:36:35.708444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:35.708394    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing" containerID="cri-o://d15639f5eb4c7111c6666e8426f397c8984a894aeb5eea230307d851f99a47b6" gracePeriod=25
Nov 02 20:36:35.818032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:35.817986    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing" containerID="cri-o://f11cad343e237e36a8cfd28988a1ad82cd13b1d8c626f1a343b22f1ace0158e1" gracePeriod=25
Nov 02 20:36:38.717956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.717918    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main" containerID="cri-o://4786637c6673084008e09c223673b7423a9270bf54b163b49ba8fdc45e46545e" gracePeriod=45
Nov 02 20:36:38.723576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.723542    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" containerID="cri-o://79d0d9a7b558f0b46a7a59052c7728ed6e511401c4e18d2666d28362b9a26e86" gracePeriod=45
Nov 02 20:36:38.803934 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.803897    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused"
Nov 02 20:36:38.894072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.894038    5099 generic.go:356] "Generic (PLEG): container finished" podID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerID="4786637c6673084008e09c223673b7423a9270bf54b163b49ba8fdc45e46545e" exitCode=0
Nov 02 20:36:38.894277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.894116    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" event={"ID":"15125abf-f6ee-4a64-b3c8-305fe354d020","Type":"ContainerDied","Data":"4786637c6673084008e09c223673b7423a9270bf54b163b49ba8fdc45e46545e"}
Nov 02 20:36:38.896053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.895917    5099 generic.go:356] "Generic (PLEG): container finished" podID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerID="79d0d9a7b558f0b46a7a59052c7728ed6e511401c4e18d2666d28362b9a26e86" exitCode=0
Nov 02 20:36:38.896053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:38.896015    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" event={"ID":"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05","Type":"ContainerDied","Data":"79d0d9a7b558f0b46a7a59052c7728ed6e511401c4e18d2666d28362b9a26e86"}
Nov 02 20:36:39.026929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.026904    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:36:39.030482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.030458    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102675    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102726    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102763    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\" (UniqueName: \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102851    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102919    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.102960    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\" (UniqueName: \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103001    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\" (UniqueName: \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103040    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103086    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103101    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103214    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\") pod \"15125abf-f6ee-4a64-b3c8-305fe354d020\" (UID: \"15125abf-f6ee-4a64-b3c8-305fe354d020\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103260    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103315    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\") pod \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\" (UID: \"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05\") "
Nov 02 20:36:39.103387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103363    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103612    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103817    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103851    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103949    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.103972    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.104020    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.104035    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.104280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.104058    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.106029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.106002    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.106168 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.106151    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "kube-api-access-mg7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:39.111265 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.111239    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:39.111411 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.111288    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "kube-api-access-zsbx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:39.111716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.111691    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm" pod "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" (UID: "f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.113186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.113165    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs" pod "15125abf-f6ee-4a64-b3c8-305fe354d020" (UID: "15125abf-f6ee-4a64-b3c8-305fe354d020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:39.205192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205159    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205189    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbx5\" (UniqueName: \"kubernetes.io/projected/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-kube-api-access-zsbx5\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205200    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15125abf-f6ee-4a64-b3c8-305fe354d020-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205210    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7ld\" (UniqueName: \"kubernetes.io/projected/15125abf-f6ee-4a64-b3c8-305fe354d020-kube-api-access-mg7ld\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205221    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205229    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15125abf-f6ee-4a64-b3c8-305fe354d020-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205238    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.205401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.205246    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.417602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.417576    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:36:39.508866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.508829    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.508884    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.508916    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.508942    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509000    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509044    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509083    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509176    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509190    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.509369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509294    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\") pod \"0f5653aa-ceeb-42b5-9801-33527250cc56\" (UID: \"0f5653aa-ceeb-42b5-9801-33527250cc56\") "
Nov 02 20:36:39.509607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509401    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.509662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509597    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:39.509809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509772    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.509891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509816    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.509891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509832    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5653aa-ceeb-42b5-9801-33527250cc56-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.509891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509831    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.509891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.509877    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.511577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.511547    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:36:39.511577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.511559    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:39.511711 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.511669    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:39.511868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.511851    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp" pod "0f5653aa-ceeb-42b5-9801-33527250cc56" (UID: "0f5653aa-ceeb-42b5-9801-33527250cc56"). InnerVolumeSpecName "kube-api-access-gsvpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:39.562257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.562167    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:36:39.610764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610725    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.610764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610758    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.611010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610774    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.611010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610823    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvpp\" (UniqueName: \"kubernetes.io/projected/0f5653aa-ceeb-42b5-9801-33527250cc56-kube-api-access-gsvpp\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.611010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610837    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.611010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.610851    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5653aa-ceeb-42b5-9801-33527250cc56-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:39.903403 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.903370    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"
Nov 02 20:36:39.903403 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.903396    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp" event={"ID":"15125abf-f6ee-4a64-b3c8-305fe354d020","Type":"ContainerDied","Data":"7f2bac11ede1ff65541f4b831eb245e7530170671b7fe4619649f2de86a37608"}
Nov 02 20:36:39.903999 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.903448    5099 scope.go:122] "RemoveContainer" containerID="4786637c6673084008e09c223673b7423a9270bf54b163b49ba8fdc45e46545e"
Nov 02 20:36:39.905010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.904987    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"
Nov 02 20:36:39.905138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.905026    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt" event={"ID":"f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05","Type":"ContainerDied","Data":"957cd778b048ebab3371002c5a7e88754e9dfb971fb1384119da11bb33432fc0"}
Nov 02 20:36:39.907238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.907215    5099 generic.go:356] "Generic (PLEG): container finished" podID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerID="e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00" exitCode=0
Nov 02 20:36:39.907435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.907273    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"
Nov 02 20:36:39.907554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.907256    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerDied","Data":"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"}
Nov 02 20:36:39.907554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.907511    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz" event={"ID":"0f5653aa-ceeb-42b5-9801-33527250cc56","Type":"ContainerDied","Data":"11b2880f2a1da13f54822c84c0ed408d5262bdb545d29adf6ff313f9e3fc3f4d"}
Nov 02 20:36:39.916004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.915982    5099 scope.go:122] "RemoveContainer" containerID="79d0d9a7b558f0b46a7a59052c7728ed6e511401c4e18d2666d28362b9a26e86"
Nov 02 20:36:39.925495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.925475    5099 scope.go:122] "RemoveContainer" containerID="e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"
Nov 02 20:36:39.934678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.934659    5099 scope.go:122] "RemoveContainer" containerID="8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"
Nov 02 20:36:39.947384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.947358    5099 scope.go:122] "RemoveContainer" containerID="e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"
Nov 02 20:36:39.947702 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:39.947683    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00\": container with ID starting with e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00 not found: ID does not exist" containerID="e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"
Nov 02 20:36:39.947759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.947711    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00"} err="failed to get container status \"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00\": rpc error: code = NotFound desc = could not find container \"e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00\": container with ID starting with e90a9024b577c479af3c6c7fc55c7bc74bcb86a3a5bbfe1df4949cdbc0daee00 not found: ID does not exist"
Nov 02 20:36:39.947759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.947731    5099 scope.go:122] "RemoveContainer" containerID="8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"
Nov 02 20:36:39.948035 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:39.948011    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3\": container with ID starting with 8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3 not found: ID does not exist" containerID="8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"
Nov 02 20:36:39.948138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:39.948044    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3"} err="failed to get container status \"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3\": rpc error: code = NotFound desc = could not find container \"8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3\": container with ID starting with 8e7eee5996522dc52c6b8f04129165447dbaf47333b4a53e3cd2fa46317dc7b3 not found: ID does not exist"
Nov 02 20:36:40.061490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.061450    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:36:40.064558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.064531    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/premium-w3-2052e3-kserve-65bd69f764-tb4nt"]
Nov 02 20:36:40.168277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.168178    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:36:40.171109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.171078    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-40425f-openshift-default-6b65b45799-pbhpz"]
Nov 02 20:36:40.276535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.275308    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:36:40.279979 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.279948    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-w3-2052e3-kserve-6866dff59-6p5vp"]
Nov 02 20:36:40.873086 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:40.873053    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:36:41.123672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.123598    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:36:41.124426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124401    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.124426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124425    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.124575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124460    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main"
Nov 02 20:36:41.124575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124466    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main"
Nov 02 20:36:41.124575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124485    5099 cpu_manager.go:414] "containerMap: removing container" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main"
Nov 02 20:36:41.124575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124490    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main"
Nov 02 20:36:41.124709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124591    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.124709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124599    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" containerName="main"
Nov 02 20:36:41.124709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124607    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" containerName="main"
Nov 02 20:36:41.124709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.124616    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.129859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.129837    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.182955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.182917    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:36:41.184463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.184437    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b1839\"/\"kube-root-ca.crt\""
Nov 02 20:36:41.184605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.184551    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b1839\"/\"default-dockercfg-5sb2r\""
Nov 02 20:36:41.184681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.184610    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b1839\"/\"openshift-service-ca.crt\""
Nov 02 20:36:41.184745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.184717    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b1839\"/\"test-model-7b1839-kserve-self-signed-certs\""
Nov 02 20:36:41.186014 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.185990    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"]
Nov 02 20:36:41.186639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.186621    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.186639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.186639    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" containerName="istio-proxy"
Nov 02 20:36:41.190565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.190544    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.233283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233232    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.233480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233375    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233420    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233456    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977hc\" (UniqueName: \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233599    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.233666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233636    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233801    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233870    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.233939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.233923    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmwm\" (UniqueName: \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.257840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.245394    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"]
Nov 02 20:36:41.284218 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.284184    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:36:41.284404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.284221    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:36:41.286338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.286315    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5653aa-ceeb-42b5-9801-33527250cc56" path="/var/lib/kubelet/pods/0f5653aa-ceeb-42b5-9801-33527250cc56/volumes"
Nov 02 20:36:41.286793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.286769    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15125abf-f6ee-4a64-b3c8-305fe354d020" path="/var/lib/kubelet/pods/15125abf-f6ee-4a64-b3c8-305fe354d020/volumes"
Nov 02 20:36:41.287200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.287186    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05" path="/var/lib/kubelet/pods/f2b47d5a-d799-4f7f-ad0b-a96cf6d6ec05/volumes"
Nov 02 20:36:41.334947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.334911    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.334955    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977hc\" (UniqueName: \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.334982    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.335145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335003    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335084    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335129    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335179    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmwm\" (UniqueName: \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.335514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335294    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.335514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335321    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335377    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335524    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335617    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.335924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.335899    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.347830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.337750    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.347830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.337941    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.347830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.341553    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.389366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.389285    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmwm\" (UniqueName: \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\") pod \"maas-api-revoke-sub-6f714aa6-wz7rg\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.389366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.389323    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977hc\" (UniqueName: \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\") pod \"test-model-7b1839-kserve-c4c5cbd7-j2fzs\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") " pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.443125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.443084    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:41.507052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.507021    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:41.816974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.816941    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:36:41.817517 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:36:41.817490    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa9a543_4e48_4552_a2dc_9ee916c94c9d.slice/crio-1983e3f18672a9565d34c0bbe651b88fc21a7fff3e804c4f292467e4e12eef7d WatchSource:0}: Error finding container 1983e3f18672a9565d34c0bbe651b88fc21a7fff3e804c4f292467e4e12eef7d: Status 404 returned error can't find the container with id 1983e3f18672a9565d34c0bbe651b88fc21a7fff3e804c4f292467e4e12eef7d
Nov 02 20:36:41.928251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.928216    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" event={"ID":"ffa9a543-4e48-4552-a2dc-9ee916c94c9d","Type":"ContainerStarted","Data":"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"}
Nov 02 20:36:41.928382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.928271    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" event={"ID":"ffa9a543-4e48-4552-a2dc-9ee916c94c9d","Type":"ContainerStarted","Data":"1983e3f18672a9565d34c0bbe651b88fc21a7fff3e804c4f292467e4e12eef7d"}
Nov 02 20:36:41.929971 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:41.929944    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"]
Nov 02 20:36:41.931096 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:36:41.931073    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf63061_2e64_4cdc_8a09_832fabd2023f.slice/crio-0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1 WatchSource:0}: Error finding container 0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1: Status 404 returned error can't find the container with id 0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1
Nov 02 20:36:42.036884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:42.036830    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" podStartSLOduration=1.036816552 podStartE2EDuration="1.036816552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:42.035164527 +0000 UTC m=+2690.909435416" watchObservedRunningTime="2026-11-02 20:36:42.036816552 +0000 UTC m=+2690.911087440"
Nov 02 20:36:42.443890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:42.443811    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:42.935820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:42.935774    5099 generic.go:356] "Generic (PLEG): container finished" podID="5cf63061-2e64-4cdc-8a09-832fabd2023f" containerID="48154fcbc52c0f63f5e72e40969bbe6f01b65c0b1b959d13e8279c8e8a4794df" exitCode=0
Nov 02 20:36:42.935967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:42.935878    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg" event={"ID":"5cf63061-2e64-4cdc-8a09-832fabd2023f","Type":"ContainerDied","Data":"48154fcbc52c0f63f5e72e40969bbe6f01b65c0b1b959d13e8279c8e8a4794df"}
Nov 02 20:36:42.935967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:42.935933    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg" event={"ID":"5cf63061-2e64-4cdc-8a09-832fabd2023f","Type":"ContainerStarted","Data":"0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1"}
Nov 02 20:36:44.089189 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.089161    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:44.170923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.170889    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\") pod \"5cf63061-2e64-4cdc-8a09-832fabd2023f\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") "
Nov 02 20:36:44.171111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.170933    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\") pod \"5cf63061-2e64-4cdc-8a09-832fabd2023f\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") "
Nov 02 20:36:44.171111 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.171051    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\" (UniqueName: \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\") pod \"5cf63061-2e64-4cdc-8a09-832fabd2023f\" (UID: \"5cf63061-2e64-4cdc-8a09-832fabd2023f\") "
Nov 02 20:36:44.171355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.171333    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca" pod "5cf63061-2e64-4cdc-8a09-832fabd2023f" (UID: "5cf63061-2e64-4cdc-8a09-832fabd2023f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:44.171552 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.171529    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:44.173061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.173030    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca" pod "5cf63061-2e64-4cdc-8a09-832fabd2023f" (UID: "5cf63061-2e64-4cdc-8a09-832fabd2023f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:44.173174 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.173155    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm" pod "5cf63061-2e64-4cdc-8a09-832fabd2023f" (UID: "5cf63061-2e64-4cdc-8a09-832fabd2023f"). InnerVolumeSpecName "kube-api-access-7zmwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:44.272795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.272701    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf63061-2e64-4cdc-8a09-832fabd2023f-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:44.272795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.272731    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmwm\" (UniqueName: \"kubernetes.io/projected/5cf63061-2e64-4cdc-8a09-832fabd2023f-kube-api-access-7zmwm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:44.922899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.922862    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:36:44.925930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.925899    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:36:44.950577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.950544    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"
Nov 02 20:36:44.950753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.950578    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg" event={"ID":"5cf63061-2e64-4cdc-8a09-832fabd2023f","Type":"ContainerDied","Data":"0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1"}
Nov 02 20:36:44.950753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:44.950648    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ea359953b3394f90af50990dc70139672e2d7fc1109a30b46c56173850c1fc1"
Nov 02 20:36:46.096253 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:46.096212    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:36:51.443426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:51.443378    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:51.457285 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:51.457250    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:51.940535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:51.940508    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:36:51.972952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:51.972929    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:36:52.007825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:52.007793    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:36:54.585476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.585409    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main" containerID="cri-o://810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f" gracePeriod=45
Nov 02 20:36:54.854929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.854524    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:36:54.922847 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.922807    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:36:54.926415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.926385    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:36:54.929777 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.929755    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.929888 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.929868    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.929948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.929931    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.930048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930033    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\" (UniqueName: \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.930118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930053    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:54.930118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930086    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:54.930118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930097    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.930275 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930176    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\") pod \"a4804654-676a-4a35-9c20-8761ec9324e8\" (UID: \"a4804654-676a-4a35-9c20-8761ec9324e8\") "
Nov 02 20:36:54.930467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930444    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:54.930623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930600    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:54.930701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930636    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:54.930701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.930647    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:54.932151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.932128    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:36:54.932234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.932206    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "kube-api-access-zwtzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:54.932273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:54.932264    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs" pod "a4804654-676a-4a35-9c20-8761ec9324e8" (UID: "a4804654-676a-4a35-9c20-8761ec9324e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:55.020733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.020695    5099 generic.go:356] "Generic (PLEG): container finished" podID="a4804654-676a-4a35-9c20-8761ec9324e8" containerID="810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f" exitCode=0
Nov 02 20:36:55.020733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.020732    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"
Nov 02 20:36:55.021012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.020795    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" event={"ID":"a4804654-676a-4a35-9c20-8761ec9324e8","Type":"ContainerDied","Data":"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"}
Nov 02 20:36:55.021012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.020852    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd" event={"ID":"a4804654-676a-4a35-9c20-8761ec9324e8","Type":"ContainerDied","Data":"87557173afb3fdcb90220092c1c6c78378f0000cc0247f4385122786c2177d28"}
Nov 02 20:36:55.021012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.020875    5099 scope.go:122] "RemoveContainer" containerID="810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"
Nov 02 20:36:55.032497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.032476    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4804654-676a-4a35-9c20-8761ec9324e8-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:55.032497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.032498    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtzr\" (UniqueName: \"kubernetes.io/projected/a4804654-676a-4a35-9c20-8761ec9324e8-kube-api-access-zwtzr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:55.032682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.032508    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4804654-676a-4a35-9c20-8761ec9324e8-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:55.035025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.035000    5099 scope.go:122] "RemoveContainer" containerID="810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"
Nov 02 20:36:55.035326 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:36:55.035306    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f\": container with ID starting with 810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f not found: ID does not exist" containerID="810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"
Nov 02 20:36:55.035413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.035333    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f"} err="failed to get container status \"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f\": rpc error: code = NotFound desc = could not find container \"810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f\": container with ID starting with 810d118911e6cd19d0a071c3750b728dce2ff9c19502314a21289e9c4abec84f not found: ID does not exist"
Nov 02 20:36:55.178718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.178683    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:36:55.183247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.183217    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/premium-w6-982b8a-kserve-7479576cf8-xl5vd"]
Nov 02 20:36:55.261453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.261419    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" path="/var/lib/kubelet/pods/a4804654-676a-4a35-9c20-8761ec9324e8/volumes"
Nov 02 20:36:55.347632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.347558    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"]
Nov 02 20:36:55.348522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.348497    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5cf63061-2e64-4cdc-8a09-832fabd2023f" containerName="revoke-keys"
Nov 02 20:36:55.348522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.348521    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf63061-2e64-4cdc-8a09-832fabd2023f" containerName="revoke-keys"
Nov 02 20:36:55.348716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.348541    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main"
Nov 02 20:36:55.348716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.348549    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main"
Nov 02 20:36:55.349197 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.349174    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf63061-2e64-4cdc-8a09-832fabd2023f" containerName="revoke-keys"
Nov 02 20:36:55.349310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.349204    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4804654-676a-4a35-9c20-8761ec9324e8" containerName="main"
Nov 02 20:36:55.354930 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.354909    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.403559 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.403523    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"]
Nov 02 20:36:55.411051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.411015    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:36:55.411051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.411047    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:36:55.439133 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.439040    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mt8\" (UniqueName: \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.439312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.439179    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.439312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.439277    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.540190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.540154    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mt8\" (UniqueName: \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.540408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.540238    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.540408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.540296    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.541127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.541096    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.542920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.542889    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.597081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.597051    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mt8\" (UniqueName: \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\") pod \"maas-api-revoke-sub-e974a4b1-zzzf8\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.674065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.674032    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:55.969853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.969821    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:36:55.976741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.976710    5099 scope.go:122] "RemoveContainer" containerID="c45797308634c17df341f4af14f15356f457379cb1e4995ce8d79e9d4c5ec4c4"
Nov 02 20:36:55.988357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:55.988336    5099 scope.go:122] "RemoveContainer" containerID="d15639f5eb4c7111c6666e8426f397c8984a894aeb5eea230307d851f99a47b6"
Nov 02 20:36:56.034958 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.034939    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"
Nov 02 20:36:56.035074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.034972    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" event={"ID":"3a5f3706-04b1-4f15-beaf-c82e37393866","Type":"ContainerDied","Data":"d15639f5eb4c7111c6666e8426f397c8984a894aeb5eea230307d851f99a47b6"}
Nov 02 20:36:56.035074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.035037    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v" event={"ID":"3a5f3706-04b1-4f15-beaf-c82e37393866","Type":"ContainerDied","Data":"72585df5460ae8d2784b104fdfad9817a253aebbdb0169195d82c9e5fc6ec9a1"}
Nov 02 20:36:56.048709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.045637    5099 generic.go:356] "Generic (PLEG): container finished" podID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerID="f11cad343e237e36a8cfd28988a1ad82cd13b1d8c626f1a343b22f1ace0158e1" exitCode=0
Nov 02 20:36:56.048709 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.045682    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" event={"ID":"9f4b5e21-b3da-45d3-8367-a5ade1342651","Type":"ContainerDied","Data":"f11cad343e237e36a8cfd28988a1ad82cd13b1d8c626f1a343b22f1ace0158e1"}
Nov 02 20:36:56.060369 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.060145    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"]
Nov 02 20:36:56.062072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.060804    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\") pod \"3a5f3706-04b1-4f15-beaf-c82e37393866\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") "
Nov 02 20:36:56.062072 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.060947    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\" (UniqueName: \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\") pod \"3a5f3706-04b1-4f15-beaf-c82e37393866\" (UID: \"3a5f3706-04b1-4f15-beaf-c82e37393866\") "
Nov 02 20:36:56.062238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.062203    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume" pod "3a5f3706-04b1-4f15-beaf-c82e37393866" (UID: "3a5f3706-04b1-4f15-beaf-c82e37393866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:56.064632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.064602    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x" pod "3a5f3706-04b1-4f15-beaf-c82e37393866" (UID: "3a5f3706-04b1-4f15-beaf-c82e37393866"). InnerVolumeSpecName "kube-api-access-vdw9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:56.163152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.162845    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:36:56.163152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.163002    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5f3706-04b1-4f15-beaf-c82e37393866-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:56.163152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.163029    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdw9x\" (UniqueName: \"kubernetes.io/projected/3a5f3706-04b1-4f15-beaf-c82e37393866-kube-api-access-vdw9x\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:56.264442 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.264350    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\" (UniqueName: \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\") pod \"9f4b5e21-b3da-45d3-8367-a5ade1342651\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") "
Nov 02 20:36:56.264609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.264526    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\") pod \"9f4b5e21-b3da-45d3-8367-a5ade1342651\" (UID: \"9f4b5e21-b3da-45d3-8367-a5ade1342651\") "
Nov 02 20:36:56.264927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.264901    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume" pod "9f4b5e21-b3da-45d3-8367-a5ade1342651" (UID: "9f4b5e21-b3da-45d3-8367-a5ade1342651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:56.265088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.265068    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4b5e21-b3da-45d3-8367-a5ade1342651-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:56.266541 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.266521    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g" pod "9f4b5e21-b3da-45d3-8367-a5ade1342651" (UID: "9f4b5e21-b3da-45d3-8367-a5ade1342651"). InnerVolumeSpecName "kube-api-access-scn4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:56.366879 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.366843    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scn4g\" (UniqueName: \"kubernetes.io/projected/9f4b5e21-b3da-45d3-8367-a5ade1342651-kube-api-access-scn4g\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:56.470899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.470838    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:36:56.475190 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:56.475157    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2052e3-77fcc49dc5-rmf7v"]
Nov 02 20:36:57.055764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.055721    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8" event={"ID":"18258622-7d03-47ab-9695-ed6bc35db917","Type":"ContainerStarted","Data":"9b03b1450842311c4d7c76025909a979d1610ab47f586dc7a173f401b4e6b5f9"}
Nov 02 20:36:57.056295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.055814    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8" event={"ID":"18258622-7d03-47ab-9695-ed6bc35db917","Type":"ContainerStarted","Data":"640cabc00f2ff46a7d042366fa12f62613949265cf92f3bacd88ec7424b545cd"}
Nov 02 20:36:57.058085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.058060    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"
Nov 02 20:36:57.058234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.058086    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx" event={"ID":"9f4b5e21-b3da-45d3-8367-a5ade1342651","Type":"ContainerDied","Data":"3aad548c541ef757f1200122f11fc15435495bf5afb8418dbe0197e867006788"}
Nov 02 20:36:57.058234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.058152    5099 scope.go:122] "RemoveContainer" containerID="f11cad343e237e36a8cfd28988a1ad82cd13b1d8c626f1a343b22f1ace0158e1"
Nov 02 20:36:57.163343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.163283    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8" podStartSLOduration=2.163266351 podStartE2EDuration="2.163266351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:57.162943032 +0000 UTC m=+2706.037213923" watchObservedRunningTime="2026-11-02 20:36:57.163266351 +0000 UTC m=+2706.037537241"
Nov 02 20:36:57.261921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.261840    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" path="/var/lib/kubelet/pods/3a5f3706-04b1-4f15-beaf-c82e37393866/volumes"
Nov 02 20:36:57.270913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.270872    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:36:57.275261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:57.275231    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2052e3-66cff889b7-hgjbx"]
Nov 02 20:36:58.070344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:58.070314    5099 generic.go:356] "Generic (PLEG): container finished" podID="18258622-7d03-47ab-9695-ed6bc35db917" containerID="9b03b1450842311c4d7c76025909a979d1610ab47f586dc7a173f401b4e6b5f9" exitCode=0
Nov 02 20:36:58.070752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:58.070404    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8" event={"ID":"18258622-7d03-47ab-9695-ed6bc35db917","Type":"ContainerDied","Data":"9b03b1450842311c4d7c76025909a979d1610ab47f586dc7a173f401b4e6b5f9"}
Nov 02 20:36:59.233506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.233470    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:36:59.258186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.258149    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" path="/var/lib/kubelet/pods/9f4b5e21-b3da-45d3-8367-a5ade1342651/volumes"
Nov 02 20:36:59.301651 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.301609    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\" (UniqueName: \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\") pod \"18258622-7d03-47ab-9695-ed6bc35db917\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") "
Nov 02 20:36:59.301853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.301714    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\") pod \"18258622-7d03-47ab-9695-ed6bc35db917\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") "
Nov 02 20:36:59.301853 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.301766    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\") pod \"18258622-7d03-47ab-9695-ed6bc35db917\" (UID: \"18258622-7d03-47ab-9695-ed6bc35db917\") "
Nov 02 20:36:59.302301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.302268    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca" pod "18258622-7d03-47ab-9695-ed6bc35db917" (UID: "18258622-7d03-47ab-9695-ed6bc35db917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:36:59.304392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.304364    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca" pod "18258622-7d03-47ab-9695-ed6bc35db917" (UID: "18258622-7d03-47ab-9695-ed6bc35db917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:36:59.304485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.304397    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8" pod "18258622-7d03-47ab-9695-ed6bc35db917" (UID: "18258622-7d03-47ab-9695-ed6bc35db917"). InnerVolumeSpecName "kube-api-access-w8mt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:36:59.404162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.404110    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8mt8\" (UniqueName: \"kubernetes.io/projected/18258622-7d03-47ab-9695-ed6bc35db917-kube-api-access-w8mt8\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:59.404162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.404159    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:36:59.404162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:36:59.404173    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18258622-7d03-47ab-9695-ed6bc35db917-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:37:00.085446 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:37:00.085415    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"
Nov 02 20:37:00.085605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:37:00.085443    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8" event={"ID":"18258622-7d03-47ab-9695-ed6bc35db917","Type":"ContainerDied","Data":"640cabc00f2ff46a7d042366fa12f62613949265cf92f3bacd88ec7424b545cd"}
Nov 02 20:37:00.085605 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:37:00.085493    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640cabc00f2ff46a7d042366fa12f62613949265cf92f3bacd88ec7424b545cd"
Nov 02 20:37:16.094724 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:37:16.094684    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:37:46.093743 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:37:46.093652    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:38:16.095857 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:38:16.095816    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:38:46.094240 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:38:46.094202    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:39:16.094884 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:39:16.094764    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:39:46.095368 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:39:46.095328    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:40:04.784564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:04.784530    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:40:16.095561 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:40:16.095527    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:40:16.831763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:16.831719    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:40:16.942445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:16.942402    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:40:19.805366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:19.805300    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerName="main" containerID="cri-o://1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158" gracePeriod=45
Nov 02 20:40:20.106515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.106490    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:40:20.264165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.263752    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.264165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.263858    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\" (UniqueName: \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.264165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.264084    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:20.264512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.264336    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.264512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.264395    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.264512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.264492    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.264705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.264528    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\") pod \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\" (UID: \"ffa9a543-4e48-4552-a2dc-9ee916c94c9d\") "
Nov 02 20:40:20.265064 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.265033    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.265596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.265569    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:20.266169 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.266142    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:20.268494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.268462    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:40:20.269898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.269871    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:40:20.270571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.270546    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc" pod "ffa9a543-4e48-4552-a2dc-9ee916c94c9d" (UID: "ffa9a543-4e48-4552-a2dc-9ee916c94c9d"). InnerVolumeSpecName "kube-api-access-977hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:20.366043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.366007    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.366043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.366036    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.366252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.366050    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.366252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.366061    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.366252 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.366075    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-977hc\" (UniqueName: \"kubernetes.io/projected/ffa9a543-4e48-4552-a2dc-9ee916c94c9d-kube-api-access-977hc\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:20.389600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.389564    5099 generic.go:356] "Generic (PLEG): container finished" podID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerID="1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158" exitCode=0
Nov 02 20:40:20.389600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.389602    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" event={"ID":"ffa9a543-4e48-4552-a2dc-9ee916c94c9d","Type":"ContainerDied","Data":"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"}
Nov 02 20:40:20.389878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.389608    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"
Nov 02 20:40:20.389878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.389627    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" event={"ID":"ffa9a543-4e48-4552-a2dc-9ee916c94c9d","Type":"ContainerDied","Data":"1983e3f18672a9565d34c0bbe651b88fc21a7fff3e804c4f292467e4e12eef7d"}
Nov 02 20:40:20.389878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.389647    5099 scope.go:122] "RemoveContainer" containerID="1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"
Nov 02 20:40:20.400288 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.400263    5099 scope.go:122] "RemoveContainer" containerID="1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"
Nov 02 20:40:20.400596 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:40:20.400575    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158\": container with ID starting with 1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158 not found: ID does not exist" containerID="1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"
Nov 02 20:40:20.400653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.400605    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158"} err="failed to get container status \"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158\": rpc error: code = NotFound desc = could not find container \"1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158\": container with ID starting with 1d4974de196e07c0c945d87668a51dd323f6c260197f441a2fc83b664879e158 not found: ID does not exist"
Nov 02 20:40:20.544623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.544591    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:40:20.548065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:20.548043    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs"]
Nov 02 20:40:21.007762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:21.007719    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7b1839/test-model-7b1839-kserve-c4c5cbd7-j2fzs" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.239:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:40:21.258232 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:21.258154    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" path="/var/lib/kubelet/pods/ffa9a543-4e48-4552-a2dc-9ee916c94c9d/volumes"
Nov 02 20:40:21.833013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:21.832970    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" podUID="42f3557f-7978-46c2-aa99-c6ecb3384492" containerName="payload-processing" containerID="cri-o://61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085" gracePeriod=25
Nov 02 20:40:21.943733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:21.943681    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing" containerID="cri-o://2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b" gracePeriod=25
Nov 02 20:40:22.107769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.107741    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:40:22.284749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.284710    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\" (UniqueName: \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\") pod \"42f3557f-7978-46c2-aa99-c6ecb3384492\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") "
Nov 02 20:40:22.284979 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.284772    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\") pod \"42f3557f-7978-46c2-aa99-c6ecb3384492\" (UID: \"42f3557f-7978-46c2-aa99-c6ecb3384492\") "
Nov 02 20:40:22.285247 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.285220    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume" pod "42f3557f-7978-46c2-aa99-c6ecb3384492" (UID: "42f3557f-7978-46c2-aa99-c6ecb3384492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:40:22.287036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.287016    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb" pod "42f3557f-7978-46c2-aa99-c6ecb3384492" (UID: "42f3557f-7978-46c2-aa99-c6ecb3384492"). InnerVolumeSpecName "kube-api-access-jhwmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:22.386134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.386104    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhwmb\" (UniqueName: \"kubernetes.io/projected/42f3557f-7978-46c2-aa99-c6ecb3384492-kube-api-access-jhwmb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:22.386134 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.386135    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f3557f-7978-46c2-aa99-c6ecb3384492-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:22.403926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.403899    5099 generic.go:356] "Generic (PLEG): container finished" podID="42f3557f-7978-46c2-aa99-c6ecb3384492" containerID="61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085" exitCode=0
Nov 02 20:40:22.404076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.403943    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"
Nov 02 20:40:22.404076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.403990    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" event={"ID":"42f3557f-7978-46c2-aa99-c6ecb3384492","Type":"ContainerDied","Data":"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"}
Nov 02 20:40:22.404076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.404038    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg" event={"ID":"42f3557f-7978-46c2-aa99-c6ecb3384492","Type":"ContainerDied","Data":"347686dc9a1406250dc129f9c253d68817c48be48919e319085748840927234c"}
Nov 02 20:40:22.404076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.404061    5099 scope.go:122] "RemoveContainer" containerID="61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"
Nov 02 20:40:22.414196 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.414177    5099 scope.go:122] "RemoveContainer" containerID="61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"
Nov 02 20:40:22.414485 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:40:22.414464    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085\": container with ID starting with 61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085 not found: ID does not exist" containerID="61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"
Nov 02 20:40:22.414573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.414491    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085"} err="failed to get container status \"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085\": rpc error: code = NotFound desc = could not find container \"61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085\": container with ID starting with 61acff336ca7c38d674088c4f89f33ea1c0571cd56f8f412e24f539cf4819085 not found: ID does not exist"
Nov 02 20:40:22.559470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.559432    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:40:22.562198 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:22.562175    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65f28f-59bc9666bc-gr5gg"]
Nov 02 20:40:23.262385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:23.262343    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f3557f-7978-46c2-aa99-c6ecb3384492" path="/var/lib/kubelet/pods/42f3557f-7978-46c2-aa99-c6ecb3384492/volumes"
Nov 02 20:40:31.415690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:31.415601    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:40:41.416178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:41.416137    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:40:42.201355 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.201287    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:40:42.306974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.306940    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\" (UniqueName: \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\") pod \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") "
Nov 02 20:40:42.307241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.307109    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\") pod \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\" (UID: \"9c448487-d84f-44fd-b7e4-a2ec1635bc62\") "
Nov 02 20:40:42.307583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.307551    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume" pod "9c448487-d84f-44fd-b7e4-a2ec1635bc62" (UID: "9c448487-d84f-44fd-b7e4-a2ec1635bc62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:40:42.309852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.309828    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6" pod "9c448487-d84f-44fd-b7e4-a2ec1635bc62" (UID: "9c448487-d84f-44fd-b7e4-a2ec1635bc62"). InnerVolumeSpecName "kube-api-access-7nkb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:40:42.409132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.409093    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c448487-d84f-44fd-b7e4-a2ec1635bc62-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:42.409132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.409126    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nkb6\" (UniqueName: \"kubernetes.io/projected/9c448487-d84f-44fd-b7e4-a2ec1635bc62-kube-api-access-7nkb6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:40:42.527392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.527299    5099 generic.go:356] "Generic (PLEG): container finished" podID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerID="2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b" exitCode=0
Nov 02 20:40:42.527392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.527362    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"
Nov 02 20:40:42.527980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.527383    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" event={"ID":"9c448487-d84f-44fd-b7e4-a2ec1635bc62","Type":"ContainerDied","Data":"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"}
Nov 02 20:40:42.527980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.527475    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn" event={"ID":"9c448487-d84f-44fd-b7e4-a2ec1635bc62","Type":"ContainerDied","Data":"55d43c37d6155cfd5f97bf89824bb76b04f37d2e6ce8bf89e379094ae6fda9b3"}
Nov 02 20:40:42.527980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.527521    5099 scope.go:122] "RemoveContainer" containerID="2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"
Nov 02 20:40:42.537582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.537556    5099 scope.go:122] "RemoveContainer" containerID="2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"
Nov 02 20:40:42.537868 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:40:42.537848    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b\": container with ID starting with 2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b not found: ID does not exist" containerID="2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"
Nov 02 20:40:42.537940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.537876    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b"} err="failed to get container status \"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b\": rpc error: code = NotFound desc = could not find container \"2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b\": container with ID starting with 2e7d5626a380bd8a80b8a7cf100fcb3d5eecd55b2d7365d9b352afd9baefe40b not found: ID does not exist"
Nov 02 20:40:42.682222 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.682184    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:40:42.687927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:42.687902    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65f28f-77c59c469f-kljkn"]
Nov 02 20:40:43.258371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:43.258330    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" path="/var/lib/kubelet/pods/9c448487-d84f-44fd-b7e4-a2ec1635bc62/volumes"
Nov 02 20:40:46.096383 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:40:46.096338    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:40:58.993040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:58.993003    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:40:58.993474 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:40:58.993272    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy" containerID="cri-o://ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398" gracePeriod=30
Nov 02 20:41:04.266045 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.265978    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:41:04.364742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364712    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.364967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364769    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.364967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364890    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.364967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364932    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.364967 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364960    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.365210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.364998    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.365210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365041    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:04.365210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365056    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:04.365210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365049    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.365210 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365130    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.365502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365249    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\") pod \"7b58d473-99cb-4c46-bb41-77c27a057533\" (UID: \"7b58d473-99cb-4c46-bb41-77c27a057533\") "
Nov 02 20:41:04.365502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365257    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:04.365502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365273    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:04.365681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365565    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:41:04.365769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365752    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.365840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365775    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.365840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365815    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.365840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365829    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.365978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.365842    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b58d473-99cb-4c46-bb41-77c27a057533-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.367673 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.367646    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:04.367820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.367723    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "kube-api-access-zjj2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:41:04.367820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.367732    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:41:04.367820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.367806    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token" pod "7b58d473-99cb-4c46-bb41-77c27a057533" (UID: "7b58d473-99cb-4c46-bb41-77c27a057533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:41:04.467274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.467235    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b58d473-99cb-4c46-bb41-77c27a057533-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.467274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.467274    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b58d473-99cb-4c46-bb41-77c27a057533-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.467488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.467289    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.467488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.467303    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjj2t\" (UniqueName: \"kubernetes.io/projected/7b58d473-99cb-4c46-bb41-77c27a057533-kube-api-access-zjj2t\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:04.708404 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.708370    5099 generic.go:356] "Generic (PLEG): container finished" podID="7b58d473-99cb-4c46-bb41-77c27a057533" containerID="ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398" exitCode=0
Nov 02 20:41:04.708583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.708426    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"
Nov 02 20:41:04.708583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.708429    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerDied","Data":"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"}
Nov 02 20:41:04.708583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.708483    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6" event={"ID":"7b58d473-99cb-4c46-bb41-77c27a057533","Type":"ContainerDied","Data":"15b53bc7459569c5ac4627e0e93bc52147e19ff7f7e0487cfce84af82a0922bb"}
Nov 02 20:41:04.708583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.708505    5099 scope.go:122] "RemoveContainer" containerID="ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"
Nov 02 20:41:04.718824 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.718772    5099 scope.go:122] "RemoveContainer" containerID="ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"
Nov 02 20:41:04.727717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.727700    5099 scope.go:122] "RemoveContainer" containerID="ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"
Nov 02 20:41:04.728002 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:04.727984    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398\": container with ID starting with ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398 not found: ID does not exist" containerID="ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"
Nov 02 20:41:04.728081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.728011    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398"} err="failed to get container status \"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398\": rpc error: code = NotFound desc = could not find container \"ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398\": container with ID starting with ce50c1b82adff0807a2dafda850091c3e5b92fddf13ca999427be6e361e30398 not found: ID does not exist"
Nov 02 20:41:04.728081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.728031    5099 scope.go:122] "RemoveContainer" containerID="ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"
Nov 02 20:41:04.728263 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:04.728243    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909\": container with ID starting with ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909 not found: ID does not exist" containerID="ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"
Nov 02 20:41:04.728317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.728269    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909"} err="failed to get container status \"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909\": rpc error: code = NotFound desc = could not find container \"ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909\": container with ID starting with ed6f0f4787ba2ca41c5a20f6f299dbc2f4d0a1757cd2614385a8933f5b2f6909 not found: ID does not exist"
Nov 02 20:41:04.867407 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.867366    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:41:04.912745 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:04.912715    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-65f28f-openshift-default-76687f7b6f-ldbt6"]
Nov 02 20:41:05.259013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:05.258979    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" path="/var/lib/kubelet/pods/7b58d473-99cb-4c46-bb41-77c27a057533/volumes"
Nov 02 20:41:16.096074 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:16.096032    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:41:41.629987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.629954    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:41:41.630540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.630153    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" podUID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" containerName="maas-api" containerID="cri-o://17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c" gracePeriod=30
Nov 02 20:41:41.912263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.912236    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:41:41.958798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.958755    5099 generic.go:356] "Generic (PLEG): container finished" podID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" containerID="17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c" exitCode=0
Nov 02 20:41:41.958963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.958820    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"
Nov 02 20:41:41.958963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.958848    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" event={"ID":"1cc88d7b-4f31-4b8a-9182-d2baf064d074","Type":"ContainerDied","Data":"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"}
Nov 02 20:41:41.958963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.958905    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4" event={"ID":"1cc88d7b-4f31-4b8a-9182-d2baf064d074","Type":"ContainerDied","Data":"6cebabb8afc24e86cde3f05934cc3d8e4061482d249cc83eac5ff07fc18ef443"}
Nov 02 20:41:41.958963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.958931    5099 scope.go:122] "RemoveContainer" containerID="17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"
Nov 02 20:41:41.968432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.968416    5099 scope.go:122] "RemoveContainer" containerID="17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"
Nov 02 20:41:41.968670 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:41.968651    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c\": container with ID starting with 17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c not found: ID does not exist" containerID="17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"
Nov 02 20:41:41.968722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:41.968678    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c"} err="failed to get container status \"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c\": rpc error: code = NotFound desc = could not find container \"17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c\": container with ID starting with 17a3e06ea321d39eb022846ae4909d33fa1eeee3663101b20c5f3b87d1ab4f3c not found: ID does not exist"
Nov 02 20:41:42.002340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.001414    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\" (UniqueName: \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\") pod \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") "
Nov 02 20:41:42.002340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.001489    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\") pod \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") "
Nov 02 20:41:42.002340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.001578    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\") pod \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") "
Nov 02 20:41:42.002340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.001691    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\") pod \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\" (UID: \"1cc88d7b-4f31-4b8a-9182-d2baf064d074\") "
Nov 02 20:41:42.013997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.003207    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca" pod "1cc88d7b-4f31-4b8a-9182-d2baf064d074" (UID: "1cc88d7b-4f31-4b8a-9182-d2baf064d074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:41:42.017886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.017771    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls" pod "1cc88d7b-4f31-4b8a-9182-d2baf064d074" (UID: "1cc88d7b-4f31-4b8a-9182-d2baf064d074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:41:42.018184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.018004    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs" pod "1cc88d7b-4f31-4b8a-9182-d2baf064d074" (UID: "1cc88d7b-4f31-4b8a-9182-d2baf064d074"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:41:42.018336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.018309    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh" pod "1cc88d7b-4f31-4b8a-9182-d2baf064d074" (UID: "1cc88d7b-4f31-4b8a-9182-d2baf064d074"). InnerVolumeSpecName "kube-api-access-5s7lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:41:42.103059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.103018    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:42.103059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.103051    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7lh\" (UniqueName: \"kubernetes.io/projected/1cc88d7b-4f31-4b8a-9182-d2baf064d074-kube-api-access-5s7lh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:42.103059 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.103063    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc88d7b-4f31-4b8a-9182-d2baf064d074-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:42.103311 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.103072    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cc88d7b-4f31-4b8a-9182-d2baf064d074-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:42.374606 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.374561    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:41:42.379701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:42.379676    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b1839-854d6dd8bc-8mzx4"]
Nov 02 20:41:43.257871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:43.257835    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" path="/var/lib/kubelet/pods/1cc88d7b-4f31-4b8a-9182-d2baf064d074/volumes"
Nov 02 20:41:45.046437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:45.046403    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"]
Nov 02 20:41:45.094358 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:45.094326    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f714aa6-wz7rg"]
Nov 02 20:41:45.257244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:45.257216    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf63061-2e64-4cdc-8a09-832fabd2023f" path="/var/lib/kubelet/pods/5cf63061-2e64-4cdc-8a09-832fabd2023f/volumes"
Nov 02 20:41:46.094216 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:46.094180    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:41:50.128859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:50.128823    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:41:50.129390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:50.129045    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy" containerID="cri-o://28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e" gracePeriod=30
Nov 02 20:41:51.980061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:51.980031    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:41:52.013423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:52.013397    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:41:55.418125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.418099    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:41:55.475419 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475381    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.475615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475432    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.475615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475495    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.475615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475522    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.475615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475563    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.475890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475842    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:55.475964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475882    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:55.475964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.475918    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:55.476078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476003    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:41:55.476131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476080    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.476131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476110    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.476240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476164    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.476240 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476209    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\") pod \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\" (UID: \"1611b8bf-6b4e-437f-8fe3-ca6681f22722\") "
Nov 02 20:41:55.476819 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476733    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:55.476957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476825    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.476957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476848    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.476957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476864    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.476957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.476878    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.478273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.478238    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:41:55.478387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.478366    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:41:55.478434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.478393    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:41:55.478969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.478948    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj" pod "1611b8bf-6b4e-437f-8fe3-ca6681f22722" (UID: "1611b8bf-6b4e-437f-8fe3-ca6681f22722"). InnerVolumeSpecName "kube-api-access-cllpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:41:55.578494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.578460    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.578494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.578489    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.578494 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.578501    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cllpj\" (UniqueName: \"kubernetes.io/projected/1611b8bf-6b4e-437f-8fe3-ca6681f22722-kube-api-access-cllpj\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.578741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.578511    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611b8bf-6b4e-437f-8fe3-ca6681f22722-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:55.578741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:55.578520    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611b8bf-6b4e-437f-8fe3-ca6681f22722-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:41:56.060401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.060361    5099 generic.go:356] "Generic (PLEG): container finished" podID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerID="28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e" exitCode=0
Nov 02 20:41:56.060612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.060421    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"
Nov 02 20:41:56.060612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.060450    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerDied","Data":"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"}
Nov 02 20:41:56.060612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.060543    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz" event={"ID":"1611b8bf-6b4e-437f-8fe3-ca6681f22722","Type":"ContainerDied","Data":"0e50bce301b37455687e832d0efac88f2f347d1360f98d6cf5a932c014181263"}
Nov 02 20:41:56.060612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.060585    5099 scope.go:122] "RemoveContainer" containerID="28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"
Nov 02 20:41:56.074613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.074596    5099 scope.go:122] "RemoveContainer" containerID="6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"
Nov 02 20:41:56.085539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.085518    5099 scope.go:122] "RemoveContainer" containerID="28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"
Nov 02 20:41:56.085857 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:56.085830    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e\": container with ID starting with 28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e not found: ID does not exist" containerID="28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"
Nov 02 20:41:56.085949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.085870    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e"} err="failed to get container status \"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e\": rpc error: code = NotFound desc = could not find container \"28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e\": container with ID starting with 28146be52b0d5323e0932de728c807c0f70198874f9ee78c3db474bd0b6bac7e not found: ID does not exist"
Nov 02 20:41:56.085949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.085895    5099 scope.go:122] "RemoveContainer" containerID="6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"
Nov 02 20:41:56.086181 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:41:56.086153    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7\": container with ID starting with 6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7 not found: ID does not exist" containerID="6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"
Nov 02 20:41:56.086228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.086188    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7"} err="failed to get container status \"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7\": rpc error: code = NotFound desc = could not find container \"6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7\": container with ID starting with 6630005f3c460e6f9d0d902cf31bba76525ba22227e3ca84b63f3e0eb3f540e7 not found: ID does not exist"
Nov 02 20:41:56.266646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.266605    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:41:56.311676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:56.311595    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b1839-openshift-default-f8c4499c-6skmz"]
Nov 02 20:41:57.261969 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:41:57.261873    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" path="/var/lib/kubelet/pods/1611b8bf-6b4e-437f-8fe3-ca6681f22722/volumes"
Nov 02 20:42:00.065306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:00.065212    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"]
Nov 02 20:42:00.067671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:00.067641    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e974a4b1-zzzf8"]
Nov 02 20:42:01.258415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:01.258379    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18258622-7d03-47ab-9695-ed6bc35db917" path="/var/lib/kubelet/pods/18258622-7d03-47ab-9695-ed6bc35db917/volumes"
Nov 02 20:42:01.802550 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:01.802511    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:42:01.918939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:01.913628    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:42:06.803744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:06.803676    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing" containerID="cri-o://76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710" gracePeriod=25
Nov 02 20:42:06.914591 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:06.914552    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing" containerID="cri-o://f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2" gracePeriod=25
Nov 02 20:42:06.938428 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:06.938389    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:42:07.945497 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:07.945457    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:42:16.097673 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:16.097625    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:42:16.938393 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:16.938357    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:42:17.945530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:17.945491    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:42:26.938048 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:42:26.938022    5099 logging.go:55] [core] [Channel #11296 SubChannel #11297]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.59:50052: connect: connection refused"
Nov 02 20:42:27.120207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.107988    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:42:27.146103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.145133    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\") pod \"e788066f-f2a1-4b06-9b2e-955d313f6367\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") "
Nov 02 20:42:27.146103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.145332    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\" (UniqueName: \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\") pod \"e788066f-f2a1-4b06-9b2e-955d313f6367\" (UID: \"e788066f-f2a1-4b06-9b2e-955d313f6367\") "
Nov 02 20:42:27.148699 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.148269    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v" pod "e788066f-f2a1-4b06-9b2e-955d313f6367" (UID: "e788066f-f2a1-4b06-9b2e-955d313f6367"). InnerVolumeSpecName "kube-api-access-w2q7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:27.148699 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.148637    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume" pod "e788066f-f2a1-4b06-9b2e-955d313f6367" (UID: "e788066f-f2a1-4b06-9b2e-955d313f6367"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:27.215862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.215835    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:42:27.246422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.246385    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\" (UniqueName: \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\") pod \"aaf2296a-806b-4344-8d11-d4c8d9f42783\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") "
Nov 02 20:42:27.246617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.246453    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\") pod \"aaf2296a-806b-4344-8d11-d4c8d9f42783\" (UID: \"aaf2296a-806b-4344-8d11-d4c8d9f42783\") "
Nov 02 20:42:27.246972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.246935    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume" pod "aaf2296a-806b-4344-8d11-d4c8d9f42783" (UID: "aaf2296a-806b-4344-8d11-d4c8d9f42783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:27.247083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.246943    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2q7v\" (UniqueName: \"kubernetes.io/projected/e788066f-f2a1-4b06-9b2e-955d313f6367-kube-api-access-w2q7v\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:27.247083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.247004    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e788066f-f2a1-4b06-9b2e-955d313f6367-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:27.249164 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.249142    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc" pod "aaf2296a-806b-4344-8d11-d4c8d9f42783" (UID: "aaf2296a-806b-4344-8d11-d4c8d9f42783"). InnerVolumeSpecName "kube-api-access-h22rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:27.270549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.270518    5099 generic.go:356] "Generic (PLEG): container finished" podID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerID="76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710" exitCode=0
Nov 02 20:42:27.270755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.270581    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"
Nov 02 20:42:27.271155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.270926    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" event={"ID":"e788066f-f2a1-4b06-9b2e-955d313f6367","Type":"ContainerDied","Data":"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"}
Nov 02 20:42:27.271155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.271030    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw" event={"ID":"e788066f-f2a1-4b06-9b2e-955d313f6367","Type":"ContainerDied","Data":"ba21a6be3e54cb0a7c75de9874615efa6a9c0f24a58b4ba02aa9474b6a12e729"}
Nov 02 20:42:27.271155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.271085    5099 scope.go:122] "RemoveContainer" containerID="76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"
Nov 02 20:42:27.272532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.272511    5099 generic.go:356] "Generic (PLEG): container finished" podID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerID="f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2" exitCode=0
Nov 02 20:42:27.272626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.272600    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" event={"ID":"aaf2296a-806b-4344-8d11-d4c8d9f42783","Type":"ContainerDied","Data":"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"}
Nov 02 20:42:27.272701 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.272639    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" event={"ID":"aaf2296a-806b-4344-8d11-d4c8d9f42783","Type":"ContainerDied","Data":"e2d3a803d18de343171983676ec492359fb0026303cfe7939818024c9975d4af"}
Nov 02 20:42:27.272769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.272708    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"
Nov 02 20:42:27.282286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.282263    5099 scope.go:122] "RemoveContainer" containerID="76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"
Nov 02 20:42:27.282565 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:27.282547    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710\": container with ID starting with 76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710 not found: ID does not exist" containerID="76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"
Nov 02 20:42:27.282636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.282571    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710"} err="failed to get container status \"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710\": rpc error: code = NotFound desc = could not find container \"76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710\": container with ID starting with 76f72eae59311622963defd2b5ed0931d9b45167454f546b40a54e6fb8af2710 not found: ID does not exist"
Nov 02 20:42:27.282636 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.282589    5099 scope.go:122] "RemoveContainer" containerID="f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"
Nov 02 20:42:27.291144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.291127    5099 scope.go:122] "RemoveContainer" containerID="f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"
Nov 02 20:42:27.291371 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:27.291356    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2\": container with ID starting with f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2 not found: ID does not exist" containerID="f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"
Nov 02 20:42:27.291440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.291376    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2"} err="failed to get container status \"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2\": rpc error: code = NotFound desc = could not find container \"f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2\": container with ID starting with f67dea92341df1766cb1d3756b4dd659e22cf0e862d9a20aed2388811d3dc5b2 not found: ID does not exist"
Nov 02 20:42:27.348298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.348201    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22rc\" (UniqueName: \"kubernetes.io/projected/aaf2296a-806b-4344-8d11-d4c8d9f42783-kube-api-access-h22rc\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:27.348298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.348239    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf2296a-806b-4344-8d11-d4c8d9f42783-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:27.428374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.428339    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:42:27.435380 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.435354    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ae8e6e-5654c4db9c-8zqpw"]
Nov 02 20:42:27.538978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.538932    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:42:27.542624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.542595    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6"]
Nov 02 20:42:27.938029 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:27.937987    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ae8e6e-dd467b455-s59r6" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded"
Nov 02 20:42:28.587379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:28.587349    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:42:28.587867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:28.587614    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" containerID="cri-o://7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970" gracePeriod=30
Nov 02 20:42:29.257981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:29.257937    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" path="/var/lib/kubelet/pods/aaf2296a-806b-4344-8d11-d4c8d9f42783/volumes"
Nov 02 20:42:29.258488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:29.258464    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" path="/var/lib/kubelet/pods/e788066f-f2a1-4b06-9b2e-955d313f6367/volumes"
Nov 02 20:42:29.765972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:29.765929    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-mkzm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:42:29.766383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:29.766001    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:42:33.857158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.857100    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:42:33.920797 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920759    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.920968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920819    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.920968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920865    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.920968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920920    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920973    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.920995    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921016    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:33.921144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921041    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921144 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921100    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921152    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\") pod \"bf64d0f7-370d-4eca-9858-39a41186c0a1\" (UID: \"bf64d0f7-370d-4eca-9858-39a41186c0a1\") "
Nov 02 20:42:33.921436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921177    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:33.921436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921281    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:33.921436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921326    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:33.921436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921402    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:33.921700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921640    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:33.921700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921661    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:33.921700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921677    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf64d0f7-370d-4eca-9858-39a41186c0a1-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:33.921700 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921692    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:33.921890 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.921705    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:33.923066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.923038    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:33.923230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.923213    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "kube-api-access-kv5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:33.923632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.923613    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:33.923692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:33.923613    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo" pod "bf64d0f7-370d-4eca-9858-39a41186c0a1" (UID: "bf64d0f7-370d-4eca-9858-39a41186c0a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:42:34.022702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.022661    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:34.022702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.022689    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:34.022702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.022702    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf64d0f7-370d-4eca-9858-39a41186c0a1-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:34.022702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.022712    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv5rg\" (UniqueName: \"kubernetes.io/projected/bf64d0f7-370d-4eca-9858-39a41186c0a1-kube-api-access-kv5rg\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:34.317307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.317266    5099 generic.go:356] "Generic (PLEG): container finished" podID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerID="7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970" exitCode=0
Nov 02 20:42:34.317521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.317324    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"
Nov 02 20:42:34.317521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.317324    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" event={"ID":"bf64d0f7-370d-4eca-9858-39a41186c0a1","Type":"ContainerDied","Data":"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"}
Nov 02 20:42:34.317521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.317457    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2" event={"ID":"bf64d0f7-370d-4eca-9858-39a41186c0a1","Type":"ContainerDied","Data":"681a06d3d420b907ccc438d66f1b643ee199babc16188942318288733ec2ff8e"}
Nov 02 20:42:34.317521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.317486    5099 scope.go:122] "RemoveContainer" containerID="7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"
Nov 02 20:42:34.328200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.328182    5099 scope.go:122] "RemoveContainer" containerID="7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"
Nov 02 20:42:34.328478 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:34.328459    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970\": container with ID starting with 7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970 not found: ID does not exist" containerID="7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"
Nov 02 20:42:34.328522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.328484    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970"} err="failed to get container status \"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970\": rpc error: code = NotFound desc = could not find container \"7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970\": container with ID starting with 7d33571fd83ee83c173c747ad55c63cde9c9810c67c8cd265b687352ef270970 not found: ID does not exist"
Nov 02 20:42:34.473272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.473234    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:42:34.476112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:34.476084    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-mkzm2"]
Nov 02 20:42:35.259449 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:35.259414    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" path="/var/lib/kubelet/pods/bf64d0f7-370d-4eca-9858-39a41186c0a1/volumes"
Nov 02 20:42:39.095924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.095888    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:42:39.096344 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.096089    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" podUID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" containerName="maas-api" containerID="cri-o://cc21837725ef98728be93515b4f2e9c272870ebb3eb172c85340c94b53e45d8e" gracePeriod=30
Nov 02 20:42:39.353434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.353406    5099 generic.go:356] "Generic (PLEG): container finished" podID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" containerID="cc21837725ef98728be93515b4f2e9c272870ebb3eb172c85340c94b53e45d8e" exitCode=0
Nov 02 20:42:39.353581 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.353483    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" event={"ID":"00a1bbbb-f935-471a-8f65-c7a4997c1d92","Type":"ContainerDied","Data":"cc21837725ef98728be93515b4f2e9c272870ebb3eb172c85340c94b53e45d8e"}
Nov 02 20:42:39.373146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.373120    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:42:39.501807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.499808    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\") pod \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") "
Nov 02 20:42:39.501807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.499864    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\") pod \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") "
Nov 02 20:42:39.501807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.499975    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\" (UniqueName: \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\") pod \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") "
Nov 02 20:42:39.501807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.500111    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") pod \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\" (UID: \"00a1bbbb-f935-471a-8f65-c7a4997c1d92\") "
Nov 02 20:42:39.504083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.502950    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca" pod "00a1bbbb-f935-471a-8f65-c7a4997c1d92" (UID: "00a1bbbb-f935-471a-8f65-c7a4997c1d92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:39.508027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.507977    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls" pod "00a1bbbb-f935-471a-8f65-c7a4997c1d92" (UID: "00a1bbbb-f935-471a-8f65-c7a4997c1d92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:42:39.517556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.515952    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2" pod "00a1bbbb-f935-471a-8f65-c7a4997c1d92" (UID: "00a1bbbb-f935-471a-8f65-c7a4997c1d92"). InnerVolumeSpecName "kube-api-access-sk6n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:39.523876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.523837    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs" pod "00a1bbbb-f935-471a-8f65-c7a4997c1d92" (UID: "00a1bbbb-f935-471a-8f65-c7a4997c1d92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:42:39.552873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.552841    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:42:39.601341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.601238    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a1bbbb-f935-471a-8f65-c7a4997c1d92-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:39.601341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.601283    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:39.601341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.601301    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6n2\" (UniqueName: \"kubernetes.io/projected/00a1bbbb-f935-471a-8f65-c7a4997c1d92-kube-api-access-sk6n2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:39.601341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:39.601311    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a1bbbb-f935-471a-8f65-c7a4997c1d92-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:40.360776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:40.360739    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4" event={"ID":"00a1bbbb-f935-471a-8f65-c7a4997c1d92","Type":"ContainerDied","Data":"12898d07d3502bc3099e92ca321084624e44e1db9ba7e820cc7bc49dbae5b7e2"}
Nov 02 20:42:40.361280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:40.360756    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"
Nov 02 20:42:40.361280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:40.360867    5099 scope.go:122] "RemoveContainer" containerID="cc21837725ef98728be93515b4f2e9c272870ebb3eb172c85340c94b53e45d8e"
Nov 02 20:42:40.516341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:40.516308    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:42:40.522075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:40.522045    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d9ec0-7b89844b6f-8wmr4"]
Nov 02 20:42:41.257011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:41.256976    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" path="/var/lib/kubelet/pods/00a1bbbb-f935-471a-8f65-c7a4997c1d92/volumes"
Nov 02 20:42:44.553160 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.553116    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" podUID="7295f041-0cc0-424b-8de6-707167639992" containerName="payload-processing" containerID="cri-o://c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34" gracePeriod=25
Nov 02 20:42:44.827617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.827591    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:42:44.957317 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.957265    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\" (UniqueName: \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\") pod \"7295f041-0cc0-424b-8de6-707167639992\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") "
Nov 02 20:42:44.957527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.957377    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\") pod \"7295f041-0cc0-424b-8de6-707167639992\" (UID: \"7295f041-0cc0-424b-8de6-707167639992\") "
Nov 02 20:42:44.958312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.958284    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume" pod "7295f041-0cc0-424b-8de6-707167639992" (UID: "7295f041-0cc0-424b-8de6-707167639992"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:44.960394 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:44.960359    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm" pod "7295f041-0cc0-424b-8de6-707167639992" (UID: "7295f041-0cc0-424b-8de6-707167639992"). InnerVolumeSpecName "kube-api-access-78dmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:45.058554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.058471    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dmm\" (UniqueName: \"kubernetes.io/projected/7295f041-0cc0-424b-8de6-707167639992-kube-api-access-78dmm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:45.058554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.058500    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7295f041-0cc0-424b-8de6-707167639992-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:45.394954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.394922    5099 generic.go:356] "Generic (PLEG): container finished" podID="7295f041-0cc0-424b-8de6-707167639992" containerID="c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34" exitCode=0
Nov 02 20:42:45.395137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.394964    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"
Nov 02 20:42:45.395137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.395005    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" event={"ID":"7295f041-0cc0-424b-8de6-707167639992","Type":"ContainerDied","Data":"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"}
Nov 02 20:42:45.395137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.395060    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp" event={"ID":"7295f041-0cc0-424b-8de6-707167639992","Type":"ContainerDied","Data":"ac99f4b7c3dea5e68439d08ff3e3c5f4ef8da8546bfbcffd29bce2a0b53b1791"}
Nov 02 20:42:45.395137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.395083    5099 scope.go:122] "RemoveContainer" containerID="c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"
Nov 02 20:42:45.405205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.405185    5099 scope.go:122] "RemoveContainer" containerID="c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"
Nov 02 20:42:45.405498 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:45.405466    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34\": container with ID starting with c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34 not found: ID does not exist" containerID="c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"
Nov 02 20:42:45.405561 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.405500    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34"} err="failed to get container status \"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34\": rpc error: code = NotFound desc = could not find container \"c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34\": container with ID starting with c864dfb418fe9d5b03921c7007e7d234cd72d17091b1a6fd50774f39fc789e34 not found: ID does not exist"
Nov 02 20:42:45.550631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.550581    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:42:45.597836 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:45.597775    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9ec0-7469d995c9-gfhzp"]
Nov 02 20:42:46.095834 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:46.095801    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:42:47.256925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:47.256888    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7295f041-0cc0-424b-8de6-707167639992" path="/var/lib/kubelet/pods/7295f041-0cc0-424b-8de6-707167639992/volumes"
Nov 02 20:42:49.635163 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:49.635127    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:42:49.635657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:49.635400    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy" containerID="cri-o://a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c" gracePeriod=30
Nov 02 20:42:50.597396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:50.597357    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:42:50.706696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:50.706665    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:42:52.318612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:52.318574    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:42:52.319075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:52.318765    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy" containerID="cri-o://709427eada979df363223f5a71d65e70c9fa4cf1c7c695b576f3dd67bf11056a" gracePeriod=30
Nov 02 20:42:54.909617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.909551    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:42:54.968805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968750    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.968805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968806    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968827    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968877    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968966    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.968998    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969058    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969075    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:54.969339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969121    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969186    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\") pod \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\" (UID: \"e3473908-6129-4fcc-b8b5-b7424e8d4c67\") "
Nov 02 20:42:54.969681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969184    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:54.969681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969440    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:54.969681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969509    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:54.969681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969619    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:54.969681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.969662    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:54.971341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.971310    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:42:54.971577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.971547    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:54.971706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.971652    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:54.971815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:54.971757    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54" pod "e3473908-6129-4fcc-b8b5-b7424e8d4c67" (UID: "e3473908-6129-4fcc-b8b5-b7424e8d4c67"). InnerVolumeSpecName "kube-api-access-fhb54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:55.071026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.070987    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071024    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071040    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071053    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhb54\" (UniqueName: \"kubernetes.io/projected/e3473908-6129-4fcc-b8b5-b7424e8d4c67-kube-api-access-fhb54\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071068    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071080    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071093    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3473908-6129-4fcc-b8b5-b7424e8d4c67-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.071341 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.071107    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3473908-6129-4fcc-b8b5-b7424e8d4c67-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:55.456291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.456250    5099 generic.go:356] "Generic (PLEG): container finished" podID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerID="a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c" exitCode=0
Nov 02 20:42:55.456490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.456297    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"
Nov 02 20:42:55.456490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.456329    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerDied","Data":"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"}
Nov 02 20:42:55.456490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.456396    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj" event={"ID":"e3473908-6129-4fcc-b8b5-b7424e8d4c67","Type":"ContainerDied","Data":"41adfc7f8230a43fdf11d7740da6d818857fec8ca8f01fc0790c1bbbbe38b53d"}
Nov 02 20:42:55.456490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.456422    5099 scope.go:122] "RemoveContainer" containerID="a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"
Nov 02 20:42:55.469923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.469893    5099 scope.go:122] "RemoveContainer" containerID="96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"
Nov 02 20:42:55.479725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.479706    5099 scope.go:122] "RemoveContainer" containerID="a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"
Nov 02 20:42:55.480050 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:55.480030    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c\": container with ID starting with a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c not found: ID does not exist" containerID="a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"
Nov 02 20:42:55.480142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.480064    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c"} err="failed to get container status \"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c\": rpc error: code = NotFound desc = could not find container \"a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c\": container with ID starting with a813901af5ef73ded9cbba48e6dad6defc0687e5ee0f9fe04763bb7993593e6c not found: ID does not exist"
Nov 02 20:42:55.480142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.480094    5099 scope.go:122] "RemoveContainer" containerID="96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"
Nov 02 20:42:55.480395 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:42:55.480370    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822\": container with ID starting with 96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822 not found: ID does not exist" containerID="96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"
Nov 02 20:42:55.480440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.480405    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822"} err="failed to get container status \"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822\": rpc error: code = NotFound desc = could not find container \"96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822\": container with ID starting with 96fa422cb9d0351cee50ba997973afaff52532803db78ab1415195cc650ac822 not found: ID does not exist"
Nov 02 20:42:55.598108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.597999    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing" containerID="cri-o://7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7" gracePeriod=25
Nov 02 20:42:55.625188 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.625151    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:42:55.646523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.646486    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:42:55.671982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.671950    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-hhzjj"]
Nov 02 20:42:55.707558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.707500    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" podUID="94eb361c-ab36-4bda-be4a-01a99f73a620" containerName="payload-pre-processing" containerID="cri-o://780d4c275bdbca8cd2fe0deefe1bbc2f8e54dfbef44f2246f610f54bc522338c" gracePeriod=25
Nov 02 20:42:55.968905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:55.968876    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:42:56.081329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.081291    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\" (UniqueName: \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\") pod \"94eb361c-ab36-4bda-be4a-01a99f73a620\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") "
Nov 02 20:42:56.081544 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.081473    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\") pod \"94eb361c-ab36-4bda-be4a-01a99f73a620\" (UID: \"94eb361c-ab36-4bda-be4a-01a99f73a620\") "
Nov 02 20:42:56.082200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.082165    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume" pod "94eb361c-ab36-4bda-be4a-01a99f73a620" (UID: "94eb361c-ab36-4bda-be4a-01a99f73a620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:56.083625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.083596    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt" pod "94eb361c-ab36-4bda-be4a-01a99f73a620" (UID: "94eb361c-ab36-4bda-be4a-01a99f73a620"). InnerVolumeSpecName "kube-api-access-jlwzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:56.183205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.183122    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb361c-ab36-4bda-be4a-01a99f73a620-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.183205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.183152    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwzt\" (UniqueName: \"kubernetes.io/projected/94eb361c-ab36-4bda-be4a-01a99f73a620-kube-api-access-jlwzt\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:56.248089 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.248059    5099 scope.go:122] "RemoveContainer" containerID="9b03b1450842311c4d7c76025909a979d1610ab47f586dc7a173f401b4e6b5f9"
Nov 02 20:42:56.257674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.257649    5099 scope.go:122] "RemoveContainer" containerID="48154fcbc52c0f63f5e72e40969bbe6f01b65c0b1b959d13e8279c8e8a4794df"
Nov 02 20:42:56.265770 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.265747    5099 scope.go:122] "RemoveContainer" containerID="780d4c275bdbca8cd2fe0deefe1bbc2f8e54dfbef44f2246f610f54bc522338c"
Nov 02 20:42:56.462554 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.462477    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"
Nov 02 20:42:56.462708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.462479    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" event={"ID":"94eb361c-ab36-4bda-be4a-01a99f73a620","Type":"ContainerDied","Data":"780d4c275bdbca8cd2fe0deefe1bbc2f8e54dfbef44f2246f610f54bc522338c"}
Nov 02 20:42:56.462708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.462602    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd" event={"ID":"94eb361c-ab36-4bda-be4a-01a99f73a620","Type":"ContainerDied","Data":"f78b9f645b69daa5737877d83569dfabb09449f1b46d3548718340e03ed2a9e0"}
Nov 02 20:42:56.675948 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.675919    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:42:56.681973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:56.681942    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb78ba-6d8bfbd4c-7lkrd"]
Nov 02 20:42:57.257063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.256985    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94eb361c-ab36-4bda-be4a-01a99f73a620" path="/var/lib/kubelet/pods/94eb361c-ab36-4bda-be4a-01a99f73a620/volumes"
Nov 02 20:42:57.257485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.257470    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" path="/var/lib/kubelet/pods/e3473908-6129-4fcc-b8b5-b7424e8d4c67/volumes"
Nov 02 20:42:57.471262 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.471229    5099 generic.go:356] "Generic (PLEG): container finished" podID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerID="709427eada979df363223f5a71d65e70c9fa4cf1c7c695b576f3dd67bf11056a" exitCode=0
Nov 02 20:42:57.471463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.471302    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerDied","Data":"709427eada979df363223f5a71d65e70c9fa4cf1c7c695b576f3dd67bf11056a"}
Nov 02 20:42:57.471463 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.471377    5099 scope.go:122] "RemoveContainer" containerID="6af8800b8c098f31670f1eb0da097daffd5df138a3088641c7fe0fba220b605e"
Nov 02 20:42:57.591448 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.591422    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:42:57.698992 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.698954    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699011    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699117    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699191 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699151    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699213    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699263    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699283    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699329    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699344    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:57.699388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699365    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\") pod \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\" (UID: \"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69\") "
Nov 02 20:42:57.699710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699397    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:57.699710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699468    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:57.699829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699722    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:57.699829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699744    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.699829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699749    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:42:57.699829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699769    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.699829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.699800    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.701624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.701595    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:42:57.701761 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.701737    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:57.701863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.701764    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "kube-api-access-ds7r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:42:57.701863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.701812    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo" pod "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" (UID: "c2328bc6-9fe4-470d-bd9e-3f494dcc1c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:42:57.800952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800916    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds7r6\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-kube-api-access-ds7r6\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.800952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800946    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.800952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800957    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.801206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800966    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.801206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800977    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:57.801206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:57.800986    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:42:58.478007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:58.477979    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"
Nov 02 20:42:58.478427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:58.478016    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg" event={"ID":"c2328bc6-9fe4-470d-bd9e-3f494dcc1c69","Type":"ContainerDied","Data":"9da3408a94c0b8d434549f7c5b4c7f58904eb2a1c60721f90bf0f76ed6c5a0cf"}
Nov 02 20:42:58.478427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:58.478074    5099 scope.go:122] "RemoveContainer" containerID="709427eada979df363223f5a71d65e70c9fa4cf1c7c695b576f3dd67bf11056a"
Nov 02 20:42:58.681744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:58.681708    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:42:58.729097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:58.729014    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0d9ec0-openshift-default-57756cbfd5-vqmgg"]
Nov 02 20:42:59.256655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:42:59.256620    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" path="/var/lib/kubelet/pods/c2328bc6-9fe4-470d-bd9e-3f494dcc1c69/volumes"
Nov 02 20:43:04.110602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:04.110566    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:43:05.646726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:05.646678    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:09.111466 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:09.111397    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing" containerID="cri-o://755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a" gracePeriod=25
Nov 02 20:43:11.443219 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:11.443173    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:11.781381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:11.781288    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:43:11.781570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:11.781490    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" podUID="cd7e92a3-3591-4d98-ba7f-73215c040874" containerName="maas-api" containerID="cri-o://707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba" gracePeriod=30
Nov 02 20:43:12.053208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.053183    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:43:12.159707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.159676    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\") pod \"cd7e92a3-3591-4d98-ba7f-73215c040874\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") "
Nov 02 20:43:12.159923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.159754    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\" (UniqueName: \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\") pod \"cd7e92a3-3591-4d98-ba7f-73215c040874\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") "
Nov 02 20:43:12.159923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.159807    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\") pod \"cd7e92a3-3591-4d98-ba7f-73215c040874\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") "
Nov 02 20:43:12.159923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.159836    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") pod \"cd7e92a3-3591-4d98-ba7f-73215c040874\" (UID: \"cd7e92a3-3591-4d98-ba7f-73215c040874\") "
Nov 02 20:43:12.160185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.160156    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca" pod "cd7e92a3-3591-4d98-ba7f-73215c040874" (UID: "cd7e92a3-3591-4d98-ba7f-73215c040874"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:12.162053 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.162021    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs" pod "cd7e92a3-3591-4d98-ba7f-73215c040874" (UID: "cd7e92a3-3591-4d98-ba7f-73215c040874"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:12.162155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.162083    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2" pod "cd7e92a3-3591-4d98-ba7f-73215c040874" (UID: "cd7e92a3-3591-4d98-ba7f-73215c040874"). InnerVolumeSpecName "kube-api-access-9s8c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:12.162155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.162101    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls" pod "cd7e92a3-3591-4d98-ba7f-73215c040874" (UID: "cd7e92a3-3591-4d98-ba7f-73215c040874"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:12.260513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.260470    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e92a3-3591-4d98-ba7f-73215c040874-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:12.260513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.260499    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8c2\" (UniqueName: \"kubernetes.io/projected/cd7e92a3-3591-4d98-ba7f-73215c040874-kube-api-access-9s8c2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:12.260513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.260511    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:12.260513 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.260519    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7e92a3-3591-4d98-ba7f-73215c040874-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:12.557365 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.557332    5099 generic.go:356] "Generic (PLEG): container finished" podID="cd7e92a3-3591-4d98-ba7f-73215c040874" containerID="707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba" exitCode=0
Nov 02 20:43:12.557904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.557392    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" event={"ID":"cd7e92a3-3591-4d98-ba7f-73215c040874","Type":"ContainerDied","Data":"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"}
Nov 02 20:43:12.557904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.557414    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"
Nov 02 20:43:12.557904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.557462    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd" event={"ID":"cd7e92a3-3591-4d98-ba7f-73215c040874","Type":"ContainerDied","Data":"b2b1d2f8e4787c196923e85eff650b409882294a386dd3ceef81900ec2eb27ff"}
Nov 02 20:43:12.557904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.557491    5099 scope.go:122] "RemoveContainer" containerID="707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"
Nov 02 20:43:12.568013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.567995    5099 scope.go:122] "RemoveContainer" containerID="707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"
Nov 02 20:43:12.568335 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:12.568305    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba\": container with ID starting with 707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba not found: ID does not exist" containerID="707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"
Nov 02 20:43:12.568396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.568345    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba"} err="failed to get container status \"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba\": rpc error: code = NotFound desc = could not find container \"707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba\": container with ID starting with 707b221fe9bf6d4d0c305ca49713d1a4ee3e118f3e27219577839b50d7c721ba not found: ID does not exist"
Nov 02 20:43:12.712167 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.712125    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:43:12.760469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:12.760430    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aaa3e7-bd9796dbf-dz2dd"]
Nov 02 20:43:13.257676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:13.257642    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7e92a3-3591-4d98-ba7f-73215c040874" path="/var/lib/kubelet/pods/cd7e92a3-3591-4d98-ba7f-73215c040874/volumes"
Nov 02 20:43:15.646950 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:43:15.646926    5099 logging.go:55] [core] [Channel #11400 SubChannel #11401]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.230:50052", ServerName: "10.132.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.230:50052: connect: connection refused"
Nov 02 20:43:15.859943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:15.859924    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:43:16.005496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.005459    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\" (UniqueName: \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\") pod \"97063b16-3959-4ea1-866c-219bc6460898\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") "
Nov 02 20:43:16.005678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.005629    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\") pod \"97063b16-3959-4ea1-866c-219bc6460898\" (UID: \"97063b16-3959-4ea1-866c-219bc6460898\") "
Nov 02 20:43:16.006148 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.006091    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume" pod "97063b16-3959-4ea1-866c-219bc6460898" (UID: "97063b16-3959-4ea1-866c-219bc6460898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:16.007735 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.007708    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f" pod "97063b16-3959-4ea1-866c-219bc6460898" (UID: "97063b16-3959-4ea1-866c-219bc6460898"). InnerVolumeSpecName "kube-api-access-jgs8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:16.096868 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:16.096829    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:43:16.107514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.107488    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97063b16-3959-4ea1-866c-219bc6460898-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:16.107514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.107516    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgs8f\" (UniqueName: \"kubernetes.io/projected/97063b16-3959-4ea1-866c-219bc6460898-kube-api-access-jgs8f\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:16.584490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.584457    5099 generic.go:356] "Generic (PLEG): container finished" podID="97063b16-3959-4ea1-866c-219bc6460898" containerID="7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7" exitCode=0
Nov 02 20:43:16.584693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.584509    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:43:16.584693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.584541    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" event={"ID":"97063b16-3959-4ea1-866c-219bc6460898","Type":"ContainerDied","Data":"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"}
Nov 02 20:43:16.584693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.584610    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" event={"ID":"97063b16-3959-4ea1-866c-219bc6460898","Type":"ContainerDied","Data":"f140a303b234f57d86cb30b129c733af69480f49dffb22a7a1b55bad5c708c21"}
Nov 02 20:43:16.584693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.584638    5099 scope.go:122] "RemoveContainer" containerID="7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"
Nov 02 20:43:16.594623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.594600    5099 scope.go:122] "RemoveContainer" containerID="7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"
Nov 02 20:43:16.594888 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:16.594872    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7\": container with ID starting with 7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7 not found: ID does not exist" containerID="7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"
Nov 02 20:43:16.594982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.594894    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7"} err="failed to get container status \"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7\": rpc error: code = NotFound desc = could not find container \"7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7\": container with ID starting with 7849e49c2ec9c6b79f25ee5188f6693efd20f47ab74d4cd56841a8718cb6e6f7 not found: ID does not exist"
Nov 02 20:43:16.646988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.646952    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 20:43:16.647376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.647062    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"
Nov 02 20:43:16.738569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.738532    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:43:16.744103 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:16.744064    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb78ba-85bc8557cc-4r8n8"]
Nov 02 20:43:17.257813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:17.257765    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97063b16-3959-4ea1-866c-219bc6460898" path="/var/lib/kubelet/pods/97063b16-3959-4ea1-866c-219bc6460898/volumes"
Nov 02 20:43:21.443180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.443132    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:21.452911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.452878    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"]
Nov 02 20:43:21.453450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453433    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453451    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453460    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453465    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453474    5099 cpu_manager.go:414] "containerMap: removing container" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453479    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453491    5099 cpu_manager.go:414] "containerMap: removing container" podUID="18258622-7d03-47ab-9695-ed6bc35db917" containerName="revoke-keys"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453496    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18258622-7d03-47ab-9695-ed6bc35db917" containerName="revoke-keys"
Nov 02 20:43:21.453504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453502    5099 cpu_manager.go:414] "containerMap: removing container" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453508    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453515    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" containerName="maas-api"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453519    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" containerName="maas-api"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453526    5099 cpu_manager.go:414] "containerMap: removing container" podUID="42f3557f-7978-46c2-aa99-c6ecb3384492" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453531    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f3557f-7978-46c2-aa99-c6ecb3384492" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453539    5099 cpu_manager.go:414] "containerMap: removing container" podUID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" containerName="maas-api"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453544    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" containerName="maas-api"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453555    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453559    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453566    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453570    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453582    5099 cpu_manager.go:414] "containerMap: removing container" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453587    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453594    5099 cpu_manager.go:414] "containerMap: removing container" podUID="94eb361c-ab36-4bda-be4a-01a99f73a620" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453599    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94eb361c-ab36-4bda-be4a-01a99f73a620" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453607    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453612    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453624    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerName="main"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453628    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerName="main"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453634    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453639    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453646    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7295f041-0cc0-424b-8de6-707167639992" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453650    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7295f041-0cc0-424b-8de6-707167639992" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453659    5099 cpu_manager.go:414] "containerMap: removing container" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453663    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453673    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453677    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453687    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453692    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453701    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453706    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453713    5099 cpu_manager.go:414] "containerMap: removing container" podUID="cd7e92a3-3591-4d98-ba7f-73215c040874" containerName="maas-api"
Nov 02 20:43:21.453759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453717    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7e92a3-3591-4d98-ba7f-73215c040874" containerName="maas-api"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453799    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5f3706-04b1-4f15-beaf-c82e37393866" containerName="payload-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453812    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e788066f-f2a1-4b06-9b2e-955d313f6367" containerName="payload-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453819    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453825    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b58d473-99cb-4c46-bb41-77c27a057533" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453832    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7e92a3-3591-4d98-ba7f-73215c040874" containerName="maas-api"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453838    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453843    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453848    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf64d0f7-370d-4eca-9858-39a41186c0a1" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453854    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="18258622-7d03-47ab-9695-ed6bc35db917" containerName="revoke-keys"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453859    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="97063b16-3959-4ea1-866c-219bc6460898" containerName="payload-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453864    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc88d7b-4f31-4b8a-9182-d2baf064d074" containerName="maas-api"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453870    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f3557f-7978-46c2-aa99-c6ecb3384492" containerName="payload-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453876    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a1bbbb-f935-471a-8f65-c7a4997c1d92" containerName="maas-api"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453881    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453885    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453891    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4b5e21-b3da-45d3-8367-a5ade1342651" containerName="payload-pre-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453898    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf2296a-806b-4344-8d11-d4c8d9f42783" containerName="payload-pre-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453904    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="94eb361c-ab36-4bda-be4a-01a99f73a620" containerName="payload-pre-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453912    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa9a543-4e48-4552-a2dc-9ee916c94c9d" containerName="main"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453916    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c448487-d84f-44fd-b7e4-a2ec1635bc62" containerName="payload-pre-processing"
Nov 02 20:43:21.454776 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.453922    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7295f041-0cc0-424b-8de6-707167639992" containerName="payload-processing"
Nov 02 20:43:21.457374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.457356    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.463986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.463255    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.463986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.463312    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.463986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.463402    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kvb\" (UniqueName: \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.507298 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.507263    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:43:21.508840 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.508810    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"]
Nov 02 20:43:21.552692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.552665    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:43:21.564227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.564190    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.564382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.564242    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.564382 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.564272    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kvb\" (UniqueName: \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.565107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.565074    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.566719 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.566694    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.617835 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.617802    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kvb\" (UniqueName: \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:21.772005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:21.771918    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:22.135707 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:22.135675    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"]
Nov 02 20:43:22.137166 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:43:22.137139    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ec62a2_b88a_42e8_8065_7aef9f1ac6eb.slice/crio-c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13 WatchSource:0}: Error finding container c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13: Status 404 returned error can't find the container with id c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13
Nov 02 20:43:22.622110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:22.622071    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6" event={"ID":"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb","Type":"ContainerStarted","Data":"da1cccebb5b1b5b4f516cc613fb5b98fe4eeb38e5aad682bab24f1a7dee58cec"}
Nov 02 20:43:22.622490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:22.622126    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6" event={"ID":"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb","Type":"ContainerStarted","Data":"c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13"}
Nov 02 20:43:22.730263 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:22.730164    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6" podStartSLOduration=1.730149285 podStartE2EDuration="1.730149285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:22.729194402 +0000 UTC m=+3091.603465289" watchObservedRunningTime="2026-11-02 20:43:22.730149285 +0000 UTC m=+3091.604420172"
Nov 02 20:43:23.628926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:23.628893    5099 generic.go:356] "Generic (PLEG): container finished" podID="f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" containerID="da1cccebb5b1b5b4f516cc613fb5b98fe4eeb38e5aad682bab24f1a7dee58cec" exitCode=0
Nov 02 20:43:23.629564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:23.628987    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6" event={"ID":"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb","Type":"ContainerDied","Data":"da1cccebb5b1b5b4f516cc613fb5b98fe4eeb38e5aad682bab24f1a7dee58cec"}
Nov 02 20:43:24.777905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.777878    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:24.795486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.795459    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\") pod \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") "
Nov 02 20:43:24.795589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.795508    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\" (UniqueName: \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\") pod \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") "
Nov 02 20:43:24.795589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.795558    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\") pod \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\" (UID: \"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb\") "
Nov 02 20:43:24.795825 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.795777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca" pod "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" (UID: "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:24.797482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.797457    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca" pod "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" (UID: "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:24.797640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.797612    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb" pod "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" (UID: "f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb"). InnerVolumeSpecName "kube-api-access-f8kvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:24.897453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.897376    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:24.897453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.897443    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:24.897655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:24.897461    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8kvb\" (UniqueName: \"kubernetes.io/projected/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb-kube-api-access-f8kvb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:25.640488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:25.640451    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6" event={"ID":"f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb","Type":"ContainerDied","Data":"c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13"}
Nov 02 20:43:25.640488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:25.640475    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"
Nov 02 20:43:25.640759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:25.640502    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e4685dba6c06696f226ceb9a88a34b3bd3cd3f2c84645d2ca1458fede0fe13"
Nov 02 20:43:25.961138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:25.961052    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:43:29.375370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.375304    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:43:29.440017 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.439970    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\") pod \"1a35586f-e776-4b67-b64a-d9210b875cd8\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") "
Nov 02 20:43:29.440226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.440062    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\" (UniqueName: \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\") pod \"1a35586f-e776-4b67-b64a-d9210b875cd8\" (UID: \"1a35586f-e776-4b67-b64a-d9210b875cd8\") "
Nov 02 20:43:29.440432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.440402    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume" pod "1a35586f-e776-4b67-b64a-d9210b875cd8" (UID: "1a35586f-e776-4b67-b64a-d9210b875cd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:29.442151 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.442130    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q" pod "1a35586f-e776-4b67-b64a-d9210b875cd8" (UID: "1a35586f-e776-4b67-b64a-d9210b875cd8"). InnerVolumeSpecName "kube-api-access-lv96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:29.541721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.541685    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a35586f-e776-4b67-b64a-d9210b875cd8-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:29.541721 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.541719    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv96q\" (UniqueName: \"kubernetes.io/projected/1a35586f-e776-4b67-b64a-d9210b875cd8-kube-api-access-lv96q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:29.663639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.663558    5099 generic.go:356] "Generic (PLEG): container finished" podID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerID="755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a" exitCode=0
Nov 02 20:43:29.663639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.663606    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"
Nov 02 20:43:29.663878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.663638    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" event={"ID":"1a35586f-e776-4b67-b64a-d9210b875cd8","Type":"ContainerDied","Data":"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"}
Nov 02 20:43:29.663878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.663701    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd" event={"ID":"1a35586f-e776-4b67-b64a-d9210b875cd8","Type":"ContainerDied","Data":"0712cf435e7feea6798f92a0564f82792bda272c6e576d8dc2f86bb8097c4fd9"}
Nov 02 20:43:29.663878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.663729    5099 scope.go:122] "RemoveContainer" containerID="755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"
Nov 02 20:43:29.673063 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.673036    5099 scope.go:122] "RemoveContainer" containerID="755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"
Nov 02 20:43:29.673337 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:29.673315    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a\": container with ID starting with 755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a not found: ID does not exist" containerID="755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"
Nov 02 20:43:29.673391 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.673347    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a"} err="failed to get container status \"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a\": rpc error: code = NotFound desc = could not find container \"755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a\": container with ID starting with 755a6aa87053933645e230890d40c882e4f9b136fb78839d5e6ec8195c75e41a not found: ID does not exist"
Nov 02 20:43:29.820218 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.820186    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:43:29.822097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:29.822074    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-aaa3e7-7f7b7f6675-42zgd"]
Nov 02 20:43:31.256054 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:31.256023    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" path="/var/lib/kubelet/pods/1a35586f-e776-4b67-b64a-d9210b875cd8/volumes"
Nov 02 20:43:32.977626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:32.977589    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:43:33.085044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:33.085007    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:43:37.977985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.977937    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing" containerID="cri-o://bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398" gracePeriod=25
Nov 02 20:43:37.980264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980239    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"]
Nov 02 20:43:37.980690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980669    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing"
Nov 02 20:43:37.980690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980688    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980708    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980716    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3473908-6129-4fcc-b8b5-b7424e8d4c67" containerName="istio-proxy"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980727    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980732    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980748    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" containerName="revoke-keys"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980753    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" containerName="revoke-keys"
Nov 02 20:43:37.980862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980856    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a35586f-e776-4b67-b64a-d9210b875cd8" containerName="payload-pre-processing"
Nov 02 20:43:37.981097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980869    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2328bc6-9fe4-470d-bd9e-3f494dcc1c69" containerName="istio-proxy"
Nov 02 20:43:37.981097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980876    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1611b8bf-6b4e-437f-8fe3-ca6681f22722" containerName="istio-proxy"
Nov 02 20:43:37.981097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.980882    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" containerName="revoke-keys"
Nov 02 20:43:37.985372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:37.985347    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.023508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.023483    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.023630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.023537    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.023630 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.023616    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9854d\" (UniqueName: \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.037986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.037957    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"]
Nov 02 20:43:38.082476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.082437    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:43:38.082476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.082470    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:43:38.085362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.085324    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing" containerID="cri-o://35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e" gracePeriod=25
Nov 02 20:43:38.124597 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.124561    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.124821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.124620    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.124821 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.124645    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9854d\" (UniqueName: \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.125376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.125354    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.126991 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.126973    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.179458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.179429    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9854d\" (UniqueName: \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\") pod \"maas-api-revoke-sub-62e2503c-jb7sw\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.298243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.298161    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:38.662628 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.662603    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"]
Nov 02 20:43:38.665608 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:43:38.665288    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eec3bf9_86d8_4276_a6d7_41a00076011b.slice/crio-2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652 WatchSource:0}: Error finding container 2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652: Status 404 returned error can't find the container with id 2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652
Nov 02 20:43:38.712333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:38.712301    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw" event={"ID":"7eec3bf9-86d8-4276-a6d7-41a00076011b","Type":"ContainerStarted","Data":"2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652"}
Nov 02 20:43:39.719429 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:39.719395    5099 generic.go:356] "Generic (PLEG): container finished" podID="7eec3bf9-86d8-4276-a6d7-41a00076011b" containerID="4320c3ae920fdcce82b0cf5eecd88ff5ea0e78875f9394dfd28a78d13db00cff" exitCode=0
Nov 02 20:43:39.719854 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:39.719478    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw" event={"ID":"7eec3bf9-86d8-4276-a6d7-41a00076011b","Type":"ContainerDied","Data":"4320c3ae920fdcce82b0cf5eecd88ff5ea0e78875f9394dfd28a78d13db00cff"}
Nov 02 20:43:40.494652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.494620    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:43:40.494966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.494937    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" containerID="cri-o://778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d" gracePeriod=30
Nov 02 20:43:40.870737 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.870710    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:40.959899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.959860    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\") pod \"7eec3bf9-86d8-4276-a6d7-41a00076011b\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") "
Nov 02 20:43:40.960107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.959915    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\") pod \"7eec3bf9-86d8-4276-a6d7-41a00076011b\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") "
Nov 02 20:43:40.960107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.959962    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\" (UniqueName: \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\") pod \"7eec3bf9-86d8-4276-a6d7-41a00076011b\" (UID: \"7eec3bf9-86d8-4276-a6d7-41a00076011b\") "
Nov 02 20:43:40.960295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.960268    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca" pod "7eec3bf9-86d8-4276-a6d7-41a00076011b" (UID: "7eec3bf9-86d8-4276-a6d7-41a00076011b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:40.962085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.962059    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca" pod "7eec3bf9-86d8-4276-a6d7-41a00076011b" (UID: "7eec3bf9-86d8-4276-a6d7-41a00076011b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:40.962268 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.962247    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d" pod "7eec3bf9-86d8-4276-a6d7-41a00076011b" (UID: "7eec3bf9-86d8-4276-a6d7-41a00076011b"). InnerVolumeSpecName "kube-api-access-9854d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:40.981116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:40.981086    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main" containerID="cri-o://4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b" gracePeriod=45
Nov 02 20:43:41.061543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.061454    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.061543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.061488    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eec3bf9-86d8-4276-a6d7-41a00076011b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.061543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.061502    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9854d\" (UniqueName: \"kubernetes.io/projected/7eec3bf9-86d8-4276-a6d7-41a00076011b-kube-api-access-9854d\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.225612 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.225586    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:43:41.365048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365011    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\" (UniqueName: \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.365241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365073    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.365241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365120    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.365241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365166    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.365485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365262    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.365731 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365397    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:41.365900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365477    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:41.365900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365541    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:41.365900 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.365742    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\") pod \"0257175b-5e44-485d-a684-dc8024f7a6d8\" (UID: \"0257175b-5e44-485d-a684-dc8024f7a6d8\") "
Nov 02 20:43:41.366161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.366138    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.366228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.366166    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.366228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.366181    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.367623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.367598    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "kube-api-access-crx9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:41.367759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.367740    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:41.368040 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.368016    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm" pod "0257175b-5e44-485d-a684-dc8024f7a6d8" (UID: "0257175b-5e44-485d-a684-dc8024f7a6d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:41.444230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.444193    5099 patch_prober.go:29] interesting pod/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 20:43:41.444397 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.444244    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 20:43:41.459959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.459929    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"]
Nov 02 20:43:41.460385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460372    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7eec3bf9-86d8-4276-a6d7-41a00076011b" containerName="revoke-keys"
Nov 02 20:43:41.460434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460386    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eec3bf9-86d8-4276-a6d7-41a00076011b" containerName="revoke-keys"
Nov 02 20:43:41.460434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460413    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main"
Nov 02 20:43:41.460434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460419    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main"
Nov 02 20:43:41.460535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460501    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerName="main"
Nov 02 20:43:41.460535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.460514    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eec3bf9-86d8-4276-a6d7-41a00076011b" containerName="revoke-keys"
Nov 02 20:43:41.463657 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.463637    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.467528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.467509    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0257175b-5e44-485d-a684-dc8024f7a6d8-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.467616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.467530    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crx9v\" (UniqueName: \"kubernetes.io/projected/0257175b-5e44-485d-a684-dc8024f7a6d8-kube-api-access-crx9v\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.467616 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.467540    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0257175b-5e44-485d-a684-dc8024f7a6d8-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:41.514964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.514933    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"]
Nov 02 20:43:41.568931 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.568854    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4xv\" (UniqueName: \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.569143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.569053    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.569282 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.569236    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.670102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.670012    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.670272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.670114    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4xv\" (UniqueName: \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.670272 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.670156    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.670734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.670707    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.672655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.672629    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.724527 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.724492    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4xv\" (UniqueName: \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\") pod \"maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.730985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.730956    5099 generic.go:356] "Generic (PLEG): container finished" podID="0257175b-5e44-485d-a684-dc8024f7a6d8" containerID="4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b" exitCode=0
Nov 02 20:43:41.731138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.731004    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"
Nov 02 20:43:41.731138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.731044    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" event={"ID":"0257175b-5e44-485d-a684-dc8024f7a6d8","Type":"ContainerDied","Data":"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"}
Nov 02 20:43:41.731138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.731090    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w" event={"ID":"0257175b-5e44-485d-a684-dc8024f7a6d8","Type":"ContainerDied","Data":"8038b73910f2c7e9cb984f3ce13479ed4eac45dacc3640a5b6ac3cd904381f1e"}
Nov 02 20:43:41.731138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.731111    5099 scope.go:122] "RemoveContainer" containerID="4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"
Nov 02 20:43:41.732570 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.732547    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw" event={"ID":"7eec3bf9-86d8-4276-a6d7-41a00076011b","Type":"ContainerDied","Data":"2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652"}
Nov 02 20:43:41.732664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.732574    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"
Nov 02 20:43:41.732664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.732597    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c3d4452184841cebbcde1b4c472c96da42449e8549a779dc152ae644a3b1652"
Nov 02 20:43:41.746455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.746429    5099 scope.go:122] "RemoveContainer" containerID="4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"
Nov 02 20:43:41.746775 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:41.746752    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b\": container with ID starting with 4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b not found: ID does not exist" containerID="4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"
Nov 02 20:43:41.746868 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.746797    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b"} err="failed to get container status \"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b\": rpc error: code = NotFound desc = could not find container \"4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b\": container with ID starting with 4610c96bab9991b5efb8ab3ffce79b7ea8e37aaaba913af3a4540ccfc81a2c0b not found: ID does not exist"
Nov 02 20:43:41.777141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.777108    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:41.947206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.947122    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:43:41.949008 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:41.948984    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-w4-cc64d3-kserve-77c5cc4bc6-9lq7w"]
Nov 02 20:43:42.140678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:42.140650    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"]
Nov 02 20:43:42.142353 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:43:42.142325    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2e07b1_6c1a_46d7_ae83_cc606fdfcd14.slice/crio-3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f WatchSource:0}: Error finding container 3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f: Status 404 returned error can't find the container with id 3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f
Nov 02 20:43:42.740918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:42.740823    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs" event={"ID":"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14","Type":"ContainerStarted","Data":"f734cf12fed4181e1729b766b57815fd1cfd70c52bec1b03dca3ed10d169b589"}
Nov 02 20:43:42.740918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:42.740877    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs" event={"ID":"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14","Type":"ContainerStarted","Data":"3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f"}
Nov 02 20:43:42.849471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:42.849421    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs" podStartSLOduration=1.849405838 podStartE2EDuration="1.849405838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:42.84828877 +0000 UTC m=+3111.722559658" watchObservedRunningTime="2026-11-02 20:43:42.849405838 +0000 UTC m=+3111.723676730"
Nov 02 20:43:43.257102 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:43.257068    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0257175b-5e44-485d-a684-dc8024f7a6d8" path="/var/lib/kubelet/pods/0257175b-5e44-485d-a684-dc8024f7a6d8/volumes"
Nov 02 20:43:43.747703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:43.747662    5099 generic.go:356] "Generic (PLEG): container finished" podID="0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" containerID="f734cf12fed4181e1729b766b57815fd1cfd70c52bec1b03dca3ed10d169b589" exitCode=0
Nov 02 20:43:43.747923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:43.747721    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs" event={"ID":"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14","Type":"ContainerDied","Data":"f734cf12fed4181e1729b766b57815fd1cfd70c52bec1b03dca3ed10d169b589"}
Nov 02 20:43:43.834152 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:43.834116    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:44.839878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:44.839842    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:44.904528 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:44.904500    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:45.000420 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.000384    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\") pod \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") "
Nov 02 20:43:45.000602 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.000557    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\") pod \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") "
Nov 02 20:43:45.000646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.000621    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\" (UniqueName: \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\") pod \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\" (UID: \"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14\") "
Nov 02 20:43:45.000995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.000966    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca" pod "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" (UID: "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:45.002609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.002582    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca" pod "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" (UID: "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:43:45.002749 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.002723    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv" pod "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" (UID: "0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14"). InnerVolumeSpecName "kube-api-access-rw4xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:45.101806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.101751    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.101806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.101795    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4xv\" (UniqueName: \"kubernetes.io/projected/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-kube-api-access-rw4xv\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.101806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.101806    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.736001 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.735976    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:43:45.759921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.759879    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"
Nov 02 20:43:45.759921 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.759907    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs" event={"ID":"0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14","Type":"ContainerDied","Data":"3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f"}
Nov 02 20:43:45.760093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.759957    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a817f58271c2613bf888e7abf6b7a81a2843b08d2770759eb07949e0f88a07f"
Nov 02 20:43:45.761678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.761655    5099 generic.go:356] "Generic (PLEG): container finished" podID="02ccbace-e464-4899-93c5-15f94b546071" containerID="778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d" exitCode=0
Nov 02 20:43:45.761828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.761725    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"
Nov 02 20:43:45.761828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.761749    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerDied","Data":"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"}
Nov 02 20:43:45.761828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.761814    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9" event={"ID":"02ccbace-e464-4899-93c5-15f94b546071","Type":"ContainerDied","Data":"eeef54257c316c4c32c4e08e486fbdbef53966422729d03c07df8577685969b0"}
Nov 02 20:43:45.762015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.761848    5099 scope.go:122] "RemoveContainer" containerID="778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"
Nov 02 20:43:45.770795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.770765    5099 scope.go:122] "RemoveContainer" containerID="843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"
Nov 02 20:43:45.779161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.779144    5099 scope.go:122] "RemoveContainer" containerID="778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"
Nov 02 20:43:45.779405 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:45.779384    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d\": container with ID starting with 778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d not found: ID does not exist" containerID="778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"
Nov 02 20:43:45.779476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.779411    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d"} err="failed to get container status \"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d\": rpc error: code = NotFound desc = could not find container \"778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d\": container with ID starting with 778b9e6ae6349448da33113d7dd01c6027add1044413b85b3f93801bf0b1ea1d not found: ID does not exist"
Nov 02 20:43:45.779476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.779433    5099 scope.go:122] "RemoveContainer" containerID="843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"
Nov 02 20:43:45.779677 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:45.779660    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9\": container with ID starting with 843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9 not found: ID does not exist" containerID="843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"
Nov 02 20:43:45.779744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.779681    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9"} err="failed to get container status \"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9\": rpc error: code = NotFound desc = could not find container \"843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9\": container with ID starting with 843411c7896791b2d57b15e26f493ca76538788fbb82c2f01c30f73c8f1fb9a9 not found: ID does not exist"
Nov 02 20:43:45.808303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808269    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808342    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808384    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808465 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808420    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808479    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808526    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808560    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808595    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808665    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\") pod \"02ccbace-e464-4899-93c5-15f94b546071\" (UID: \"02ccbace-e464-4899-93c5-15f94b546071\") "
Nov 02 20:43:45.808997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.808890    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:45.809052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.809028    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.809107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.809086    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:45.809460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.809413    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:45.809592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.809457    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:45.809592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.809473    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:45.811060 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.811033    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 20:43:45.811476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.811454    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:43:45.811710 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.811679    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:45.811841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.811825    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4" pod "02ccbace-e464-4899-93c5-15f94b546071" (UID: "02ccbace-e464-4899-93c5-15f94b546071"). InnerVolumeSpecName "kube-api-access-8tkp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:45.909662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909632    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.909662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909661    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909672    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909681    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909690    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ccbace-e464-4899-93c5-15f94b546071-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909698    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ccbace-e464-4899-93c5-15f94b546071-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909707    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkp4\" (UniqueName: \"kubernetes.io/projected/02ccbace-e464-4899-93c5-15f94b546071-kube-api-access-8tkp4\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.910081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.909716    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ccbace-e464-4899-93c5-15f94b546071-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:45.942435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:45.942406    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:43:46.104277 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:46.104238    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:43:46.210769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:46.210731    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:43:46.215876 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:46.215847    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2b3b9e-openshift-default-6b489557c9-bzkr9"]
Nov 02 20:43:47.259648 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:47.259612    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ccbace-e464-4899-93c5-15f94b546071" path="/var/lib/kubelet/pods/02ccbace-e464-4899-93c5-15f94b546071/volumes"
Nov 02 20:43:53.834766 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:53.834728    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:54.839363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:54.839325    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:43:56.973579 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.973537    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"]
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974019    5099 cpu_manager.go:414] "containerMap: removing container" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974034    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974061    5099 cpu_manager.go:414] "containerMap: removing container" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974067    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974076    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" containerName="revoke-keys"
Nov 02 20:43:56.974095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974081    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" containerName="revoke-keys"
Nov 02 20:43:56.974444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974162    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:43:56.974444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.974171    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" containerName="revoke-keys"
Nov 02 20:43:56.978944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:56.978922    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.028742 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.028701    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"]
Nov 02 20:43:57.073691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.073657    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:43:57.073905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.073704    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:43:57.115042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.115006    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.115248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.115163    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhp6p\" (UniqueName: \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.115334 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.115310    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.216158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.216077    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.216158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.216114    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.216343 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.216189    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhp6p\" (UniqueName: \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.216891 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.216867    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.218306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.218279    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.270674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.270639    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhp6p\" (UniqueName: \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\") pod \"maas-api-revoke-sub-f6687192-64cjn\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.292312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.292278    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:43:57.667949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.667912    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"]
Nov 02 20:43:57.669098 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:43:57.669058    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44cff28_ea8d_48bf_b65d_ef21388f66a3.slice/crio-6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3 WatchSource:0}: Error finding container 6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3: Status 404 returned error can't find the container with id 6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3
Nov 02 20:43:57.827625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:57.827593    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn" event={"ID":"c44cff28-ea8d-48bf-b65d-ef21388f66a3","Type":"ContainerStarted","Data":"6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3"}
Nov 02 20:43:58.249127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.249099    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:43:58.369500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.369474    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:43:58.428023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.427992    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\" (UniqueName: \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\") pod \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") "
Nov 02 20:43:58.428226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.428050    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\") pod \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\" (UID: \"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d\") "
Nov 02 20:43:58.428443 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.428421    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume" pod "7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" (UID: "7ed0e859-8ad2-46f7-b27e-2f30ab98e37d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:58.430115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.430084    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc" pod "7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" (UID: "7ed0e859-8ad2-46f7-b27e-2f30ab98e37d"). InnerVolumeSpecName "kube-api-access-zjwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:58.528941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.528855    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\" (UniqueName: \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\") pod \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") "
Nov 02 20:43:58.529115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.529018    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\") pod \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\" (UID: \"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e\") "
Nov 02 20:43:58.529389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.529367    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjwwc\" (UniqueName: \"kubernetes.io/projected/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-kube-api-access-zjwwc\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:58.529504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.529397    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:58.529677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.529636    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume" pod "9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" (UID: "9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:43:58.531418 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.531395    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7" pod "9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" (UID: "9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e"). InnerVolumeSpecName "kube-api-access-rx7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:43:58.630850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.630817    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7b7\" (UniqueName: \"kubernetes.io/projected/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-kube-api-access-rx7b7\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:58.630850 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.630845    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:43:58.834874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.834762    5099 generic.go:356] "Generic (PLEG): container finished" podID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerID="35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e" exitCode=0
Nov 02 20:43:58.834874 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.834849    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"
Nov 02 20:43:58.835094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.834847    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" event={"ID":"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e","Type":"ContainerDied","Data":"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"}
Nov 02 20:43:58.835094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.834996    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt" event={"ID":"9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e","Type":"ContainerDied","Data":"125dce78f3e562f1fc444b72d1ef0982e9bd50981569633efc9082756e92bdd6"}
Nov 02 20:43:58.835094 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.835033    5099 scope.go:122] "RemoveContainer" containerID="35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"
Nov 02 20:43:58.836705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.836509    5099 generic.go:356] "Generic (PLEG): container finished" podID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerID="bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398" exitCode=0
Nov 02 20:43:58.836705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.836562    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"
Nov 02 20:43:58.836705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.836607    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" event={"ID":"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d","Type":"ContainerDied","Data":"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"}
Nov 02 20:43:58.836705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.836655    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz" event={"ID":"7ed0e859-8ad2-46f7-b27e-2f30ab98e37d","Type":"ContainerDied","Data":"13640f2aab9819b6c864686730fa22578dcb15b3b3a06f54abaf7b1f28aca371"}
Nov 02 20:43:58.840221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.840194    5099 generic.go:356] "Generic (PLEG): container finished" podID="c44cff28-ea8d-48bf-b65d-ef21388f66a3" containerID="5dde0d73546b9723880a1fa1a62b9ed214da31bbf906b50e0baf8521dd27e184" exitCode=0
Nov 02 20:43:58.840388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.840229    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn" event={"ID":"c44cff28-ea8d-48bf-b65d-ef21388f66a3","Type":"ContainerDied","Data":"5dde0d73546b9723880a1fa1a62b9ed214da31bbf906b50e0baf8521dd27e184"}
Nov 02 20:43:58.845953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.845870    5099 scope.go:122] "RemoveContainer" containerID="35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"
Nov 02 20:43:58.846364 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:58.846340    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e\": container with ID starting with 35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e not found: ID does not exist" containerID="35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"
Nov 02 20:43:58.846462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.846372    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e"} err="failed to get container status \"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e\": rpc error: code = NotFound desc = could not find container \"35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e\": container with ID starting with 35200ca989a2ffacbd97fafe83e6b87db52f6edaac63f6c4b73e83598ed1999e not found: ID does not exist"
Nov 02 20:43:58.846462 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.846390    5099 scope.go:122] "RemoveContainer" containerID="bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"
Nov 02 20:43:58.854927 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.854911    5099 scope.go:122] "RemoveContainer" containerID="bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"
Nov 02 20:43:58.855183 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:43:58.855166    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398\": container with ID starting with bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398 not found: ID does not exist" containerID="bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"
Nov 02 20:43:58.855233 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.855191    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398"} err="failed to get container status \"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398\": rpc error: code = NotFound desc = could not find container \"bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398\": container with ID starting with bb83513ba3b3f7849788dc86873fcf2ee0fb91e4a26fd00ac3afdd8a6c019398 not found: ID does not exist"
Nov 02 20:43:58.993590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.993553    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:43:58.997172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:58.997143    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc64d3-6d94595c89-hlkhz"]
Nov 02 20:43:59.094329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.094244    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"]
Nov 02 20:43:59.096056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.096034    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc64d3-j64j6"]
Nov 02 20:43:59.100061 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.100039    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:43:59.103101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.103075    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc64d3-b59bbd8f5-64bkt"]
Nov 02 20:43:59.256438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.256403    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" path="/var/lib/kubelet/pods/7ed0e859-8ad2-46f7-b27e-2f30ab98e37d/volumes"
Nov 02 20:43:59.256845 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.256797    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" path="/var/lib/kubelet/pods/9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e/volumes"
Nov 02 20:43:59.257177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.257163    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb" path="/var/lib/kubelet/pods/f1ec62a2-b88a-42e8-8065-7aef9f1ac6eb/volumes"
Nov 02 20:43:59.989162 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:43:59.989134    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:44:00.149426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.149374    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\" (UniqueName: \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\") pod \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") "
Nov 02 20:44:00.149746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.149726    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\") pod \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") "
Nov 02 20:44:00.149970 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.149953    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\") pod \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\" (UID: \"c44cff28-ea8d-48bf-b65d-ef21388f66a3\") "
Nov 02 20:44:00.150985 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.150959    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca" pod "c44cff28-ea8d-48bf-b65d-ef21388f66a3" (UID: "c44cff28-ea8d-48bf-b65d-ef21388f66a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:44:00.159125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.159088    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p" pod "c44cff28-ea8d-48bf-b65d-ef21388f66a3" (UID: "c44cff28-ea8d-48bf-b65d-ef21388f66a3"). InnerVolumeSpecName "kube-api-access-mhp6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:44:00.160680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.160648    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca" pod "c44cff28-ea8d-48bf-b65d-ef21388f66a3" (UID: "c44cff28-ea8d-48bf-b65d-ef21388f66a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:44:00.251666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.251624    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhp6p\" (UniqueName: \"kubernetes.io/projected/c44cff28-ea8d-48bf-b65d-ef21388f66a3-kube-api-access-mhp6p\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:00.251666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.251660    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:00.251666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.251670    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44cff28-ea8d-48bf-b65d-ef21388f66a3-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:00.854412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.854380    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"
Nov 02 20:44:00.854841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.854408    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn" event={"ID":"c44cff28-ea8d-48bf-b65d-ef21388f66a3","Type":"ContainerDied","Data":"6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3"}
Nov 02 20:44:00.854841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.854457    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd13fc8bd48b877c7f967494b0ce57695db4d31dc06710716e6259128c813d3"
Nov 02 20:44:00.962634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:00.962591    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main" containerID="cri-o://4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25" gracePeriod=45
Nov 02 20:44:01.218939 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.218911    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:44:01.361458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361418    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361497    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\" (UniqueName: \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361541    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\" (UniqueName: \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361584    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361628    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361706    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:01.361925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361739    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\") pod \"881614d6-c2ab-4039-b7b5-7230392dfda8\" (UID: \"881614d6-c2ab-4039-b7b5-7230392dfda8\") "
Nov 02 20:44:01.361925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.361893    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:01.362485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.362180    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:01.362485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.362229    5099 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-home\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.362485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.362253    5099 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-tmp-dir\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.364085 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.364054    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:44:01.364226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.364170    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "kube-api-access-whxfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:44:01.364296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.364249    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs" pod "881614d6-c2ab-4039-b7b5-7230392dfda8" (UID: "881614d6-c2ab-4039-b7b5-7230392dfda8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:44:01.463767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.463686    5099 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-model-cache\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.463767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.463718    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxfm\" (UniqueName: \"kubernetes.io/projected/881614d6-c2ab-4039-b7b5-7230392dfda8-kube-api-access-whxfm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.463767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.463733    5099 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881614d6-c2ab-4039-b7b5-7230392dfda8-tls-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.463767 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.463744    5099 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881614d6-c2ab-4039-b7b5-7230392dfda8-dshm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:01.860726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.860697    5099 generic.go:356] "Generic (PLEG): container finished" podID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerID="4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25" exitCode=0
Nov 02 20:44:01.861145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.860746    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"
Nov 02 20:44:01.861145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.860801    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" event={"ID":"881614d6-c2ab-4039-b7b5-7230392dfda8","Type":"ContainerDied","Data":"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"}
Nov 02 20:44:01.861145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.860848    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb" event={"ID":"881614d6-c2ab-4039-b7b5-7230392dfda8","Type":"ContainerDied","Data":"1e993b6baa839d45858a0cacbdac3f03da33bfc5bf997418f068e0538267b540"}
Nov 02 20:44:01.861145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.860870    5099 scope.go:122] "RemoveContainer" containerID="4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"
Nov 02 20:44:01.870923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.870904    5099 scope.go:122] "RemoveContainer" containerID="4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"
Nov 02 20:44:01.871194 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:44:01.871171    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25\": container with ID starting with 4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25 not found: ID does not exist" containerID="4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"
Nov 02 20:44:01.871267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:01.871207    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25"} err="failed to get container status \"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25\": rpc error: code = NotFound desc = could not find container \"4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25\": container with ID starting with 4abd497636d3295afd75ea02440231e55cd67be9fd6968f3afa691ffbb918e25 not found: ID does not exist"
Nov 02 20:44:02.047822 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:02.047763    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:44:02.049652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:02.049623    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-w0-035e47-kserve-765c7dbfff-xdlgb"]
Nov 02 20:44:03.256338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:03.256306    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" path="/var/lib/kubelet/pods/881614d6-c2ab-4039-b7b5-7230392dfda8/volumes"
Nov 02 20:44:10.156392 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:10.156357    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"]
Nov 02 20:44:10.203371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:10.203337    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035e47-tnxqs"]
Nov 02 20:44:11.255637 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:11.255597    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14" path="/var/lib/kubelet/pods/0a2e07b1-6c1a-46d7-ae83-cc606fdfcd14/volumes"
Nov 02 20:44:16.093920 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:44:16.093876    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:44:45.610052 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610011    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"]
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610422    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610433    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610446    5099 cpu_manager.go:414] "containerMap: removing container" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610451    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610469    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610474    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610484    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c44cff28-ea8d-48bf-b65d-ef21388f66a3" containerName="revoke-keys"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610489    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44cff28-ea8d-48bf-b65d-ef21388f66a3" containerName="revoke-keys"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610557    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ccbace-e464-4899-93c5-15f94b546071" containerName="istio-proxy"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610565    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1aa0e1-9d3e-4523-9982-2b5a5dbdf21e" containerName="payload-pre-processing"
Nov 02 20:44:45.610568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610571    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="881614d6-c2ab-4039-b7b5-7230392dfda8" containerName="main"
Nov 02 20:44:45.610959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610580    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed0e859-8ad2-46f7-b27e-2f30ab98e37d" containerName="payload-processing"
Nov 02 20:44:45.610959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.610586    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44cff28-ea8d-48bf-b65d-ef21388f66a3" containerName="revoke-keys"
Nov 02 20:44:45.612692 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.612672    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.682214 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.682179    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.682378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.682301    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzfg\" (UniqueName: \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.682378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.682347    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.709639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.709609    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:44:45.709639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.709628    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:44:45.711609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.711589    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"]
Nov 02 20:44:45.783469 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.783444    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.783639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.783485    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzfg\" (UniqueName: \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.783639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.783505    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.784165 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.784128    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.785989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.785968    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.837106 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.837074    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzfg\" (UniqueName: \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\") pod \"maas-api-revoke-sub-85237e45-m967m\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:45.924759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:45.924671    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:46.094182 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:44:46.094145    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:44:46.287294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:46.287265    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"]
Nov 02 20:44:46.289426 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:44:46.289393    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a09819_2406_4f54_806c_f312acf2b850.slice/crio-2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9 WatchSource:0}: Error finding container 2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9: Status 404 returned error can't find the container with id 2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9
Nov 02 20:44:47.106030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:47.105995    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m" event={"ID":"f5a09819-2406-4f54-806c-f312acf2b850","Type":"ContainerStarted","Data":"a14d8eae165b2cc0ee4349150996df8f71f984d8e816f7162577fcac46ee7baa"}
Nov 02 20:44:47.106447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:47.106050    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m" event={"ID":"f5a09819-2406-4f54-806c-f312acf2b850","Type":"ContainerStarted","Data":"2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9"}
Nov 02 20:44:47.214884 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:47.214825    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m" podStartSLOduration=2.214807435 podStartE2EDuration="2.214807435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:47.213164805 +0000 UTC m=+3176.087435693" watchObservedRunningTime="2026-11-02 20:44:47.214807435 +0000 UTC m=+3176.089078324"
Nov 02 20:44:48.120639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:48.120600    5099 generic.go:356] "Generic (PLEG): container finished" podID="f5a09819-2406-4f54-806c-f312acf2b850" containerID="a14d8eae165b2cc0ee4349150996df8f71f984d8e816f7162577fcac46ee7baa" exitCode=0
Nov 02 20:44:48.121078 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:48.120689    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m" event={"ID":"f5a09819-2406-4f54-806c-f312acf2b850","Type":"ContainerDied","Data":"a14d8eae165b2cc0ee4349150996df8f71f984d8e816f7162577fcac46ee7baa"}
Nov 02 20:44:49.278609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.278586    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:49.314371 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.314342    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\") pod \"f5a09819-2406-4f54-806c-f312acf2b850\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") "
Nov 02 20:44:49.314508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.314381    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\" (UniqueName: \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\") pod \"f5a09819-2406-4f54-806c-f312acf2b850\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") "
Nov 02 20:44:49.314508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.314480    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\") pod \"f5a09819-2406-4f54-806c-f312acf2b850\" (UID: \"f5a09819-2406-4f54-806c-f312acf2b850\") "
Nov 02 20:44:49.314894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.314872    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca" pod "f5a09819-2406-4f54-806c-f312acf2b850" (UID: "f5a09819-2406-4f54-806c-f312acf2b850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:44:49.316451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.316429    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca" pod "f5a09819-2406-4f54-806c-f312acf2b850" (UID: "f5a09819-2406-4f54-806c-f312acf2b850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:44:49.316574 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.316558    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg" pod "f5a09819-2406-4f54-806c-f312acf2b850" (UID: "f5a09819-2406-4f54-806c-f312acf2b850"). InnerVolumeSpecName "kube-api-access-hxzfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:44:49.415408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.415317    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:49.415408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.415344    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxzfg\" (UniqueName: \"kubernetes.io/projected/f5a09819-2406-4f54-806c-f312acf2b850-kube-api-access-hxzfg\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:49.415408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:49.415355    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a09819-2406-4f54-806c-f312acf2b850-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:44:50.132000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:50.131966    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m" event={"ID":"f5a09819-2406-4f54-806c-f312acf2b850","Type":"ContainerDied","Data":"2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9"}
Nov 02 20:44:50.132000 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:50.132001    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"
Nov 02 20:44:50.132231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:50.132013    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad640a85af82cb53d3216ba9abdb4add1c0094f5c15f3564ba2ce2d2453eac9"
Nov 02 20:44:59.692361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.692283    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"]
Nov 02 20:44:59.692774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.692742    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f5a09819-2406-4f54-806c-f312acf2b850" containerName="revoke-keys"
Nov 02 20:44:59.692774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.692757    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a09819-2406-4f54-806c-f312acf2b850" containerName="revoke-keys"
Nov 02 20:44:59.692882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.692857    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a09819-2406-4f54-806c-f312acf2b850" containerName="revoke-keys"
Nov 02 20:44:59.697423 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.697402    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.746490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.746455    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\""
Nov 02 20:44:59.795915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.795883    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"]
Nov 02 20:44:59.812978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.812942    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqks\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-kube-api-access-lnqks\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.812983    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813011    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813084    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813130    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813192    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8143c2ae-2081-4d35-bd74-1a43b243de1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813217    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813287    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.813441 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.813333    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914629    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8143c2ae-2081-4d35-bd74-1a43b243de1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914666 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914672    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914691    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914715    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914756    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqks\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-kube-api-access-lnqks\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914913    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.914981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914949    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.914985    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915043    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915270 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915187    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915299    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915344    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915518    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.915576 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.915535    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8143c2ae-2081-4d35-bd74-1a43b243de1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.917201 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.917179    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.917437 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.917418    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.968345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.968271    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:44:59.968516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:44:59.968492    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqks\" (UniqueName: \"kubernetes.io/projected/8143c2ae-2081-4d35-bd74-1a43b243de1b-kube-api-access-lnqks\") pod \"maas-default-gateway-openshift-default-696f47c6c-vfwzd\" (UID: \"8143c2ae-2081-4d35-bd74-1a43b243de1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:00.009988 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.009947    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:00.150173 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.150128    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:00.154110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.154088    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.204417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.204376    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:45:00.205687 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.205658    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:00.217744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.217714    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr2qf\" (UniqueName: \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.217952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.217927    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.319099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.318998    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.319291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.319171    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr2qf\" (UniqueName: \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.319629 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.319609    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.373131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.373096    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr2qf\" (UniqueName: \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\") pod \"maas-api-key-cleanup-29894205-zbx9k\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.420522 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.420492    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"]
Nov 02 20:45:00.422485 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:45:00.422460    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8143c2ae_2081_4d35_bd74_1a43b243de1b.slice/crio-1b859fe725bd238c1e24622570876bf5aa4473d7ef79a69c634553dffdd9168c WatchSource:0}: Error finding container 1b859fe725bd238c1e24622570876bf5aa4473d7ef79a69c634553dffdd9168c: Status 404 returned error can't find the container with id 1b859fe725bd238c1e24622570876bf5aa4473d7ef79a69c634553dffdd9168c
Nov 02 20:45:00.424828 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.424777    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:45:00.424925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.424877    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:45:00.424925 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.424910    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:45:00.468243 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.468222    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:00.834837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:00.834701    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:00.836556 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:45:00.836529    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7564bd9e_6580_40e6_be30_2f2ca382a033.slice/crio-780be1dc49803050e9a945c74a3ad679e22aa3c116758555363d8633fa5e36b7 WatchSource:0}: Error finding container 780be1dc49803050e9a945c74a3ad679e22aa3c116758555363d8633fa5e36b7: Status 404 returned error can't find the container with id 780be1dc49803050e9a945c74a3ad679e22aa3c116758555363d8633fa5e36b7
Nov 02 20:45:01.190293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.190259    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerStarted","Data":"8904787d972d4a7412599bb06b2e0e4600883f7428d242d6ab7c92c1bf66775e"}
Nov 02 20:45:01.190430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.190315    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerStarted","Data":"780be1dc49803050e9a945c74a3ad679e22aa3c116758555363d8633fa5e36b7"}
Nov 02 20:45:01.192149 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.192120    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" event={"ID":"8143c2ae-2081-4d35-bd74-1a43b243de1b","Type":"ContainerStarted","Data":"715ccc7835788177f284478a2cc1977ebb671da2609fd4f2bb4c85be64ef7d75"}
Nov 02 20:45:01.192336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.192164    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" event={"ID":"8143c2ae-2081-4d35-bd74-1a43b243de1b","Type":"ContainerStarted","Data":"1b859fe725bd238c1e24622570876bf5aa4473d7ef79a69c634553dffdd9168c"}
Nov 02 20:45:01.304720 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.304663    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" podStartSLOduration=1.30465038 podStartE2EDuration="1.30465038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.303927203 +0000 UTC m=+3190.178198090" watchObservedRunningTime="2026-11-02 20:45:01.30465038 +0000 UTC m=+3190.178921267"
Nov 02 20:45:01.417387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:01.417336    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podStartSLOduration=2.417322109 podStartE2EDuration="2.417322109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.416496593 +0000 UTC m=+3190.290767481" watchObservedRunningTime="2026-11-02 20:45:01.417322109 +0000 UTC m=+3190.291592996"
Nov 02 20:45:02.010273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.010242    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:02.012207 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.012172    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:02.012323 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.012229    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:02.199385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.199301    5099 generic.go:356] "Generic (PLEG): container finished" podID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerID="8904787d972d4a7412599bb06b2e0e4600883f7428d242d6ab7c92c1bf66775e" exitCode=60
Nov 02 20:45:02.199569 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.199391    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerDied","Data":"8904787d972d4a7412599bb06b2e0e4600883f7428d242d6ab7c92c1bf66775e"}
Nov 02 20:45:02.199644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:02.199629    5099 scope.go:122] "RemoveContainer" containerID="8904787d972d4a7412599bb06b2e0e4600883f7428d242d6ab7c92c1bf66775e"
Nov 02 20:45:03.011255 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.011217    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:03.011757 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.011279    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:03.206500 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.206459    5099 generic.go:356] "Generic (PLEG): container finished" podID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerID="4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166" exitCode=60
Nov 02 20:45:03.206726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.206539    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerDied","Data":"4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166"}
Nov 02 20:45:03.206726 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.206609    5099 scope.go:122] "RemoveContainer" containerID="8904787d972d4a7412599bb06b2e0e4600883f7428d242d6ab7c92c1bf66775e"
Nov 02 20:45:03.206906 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:03.206891    5099 scope.go:122] "RemoveContainer" containerID="4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166"
Nov 02 20:45:03.207193 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:45:03.207166    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-zbx9k_odh-ai-gateway-infra(7564bd9e-6580-40e6-be30-2f2ca382a033)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033"
Nov 02 20:45:04.010763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:04.010724    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:04.010980 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:04.010805    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:04.213667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:04.213637    5099 scope.go:122] "RemoveContainer" containerID="4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166"
Nov 02 20:45:04.214150 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:45:04.213956    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-zbx9k_odh-ai-gateway-infra(7564bd9e-6580-40e6-be30-2f2ca382a033)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033"
Nov 02 20:45:05.010685 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:05.010650    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:05.010901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:05.010709    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:06.010598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:06.010563    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:06.011084 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:06.010618    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:07.010916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:07.010873    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 20:45:07.011312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:07.010945    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 20:45:08.757361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:08.757328    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:08.757911 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:08.757656    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:08.758656 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:08.758631    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd"
Nov 02 20:45:15.250228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:15.250192    5099 scope.go:122] "RemoveContainer" containerID="4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166"
Nov 02 20:45:16.093121 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:45:16.093086    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:45:16.283942 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:16.283912    5099 generic.go:356] "Generic (PLEG): container finished" podID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerID="aa0cd0a70441da39d2135d6004e5a6507fd41bb67bd428ab48db2c5b95894a21" exitCode=60
Nov 02 20:45:16.284387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:16.283973    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerDied","Data":"aa0cd0a70441da39d2135d6004e5a6507fd41bb67bd428ab48db2c5b95894a21"}
Nov 02 20:45:16.284387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:16.284031    5099 scope.go:122] "RemoveContainer" containerID="4870a0ccf0845023b5d8d316b81826d7998a2b935d0eea5766d78f174f319166"
Nov 02 20:45:16.284387 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:16.284338    5099 scope.go:122] "RemoveContainer" containerID="aa0cd0a70441da39d2135d6004e5a6507fd41bb67bd428ab48db2c5b95894a21"
Nov 02 20:45:16.284609 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:45:16.284585    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-zbx9k_odh-ai-gateway-infra(7564bd9e-6580-40e6-be30-2f2ca382a033)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033"
Nov 02 20:45:17.400293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.400257    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:17.538131 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.538108    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:17.614032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.614000    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\" (UniqueName: \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\") pod \"7564bd9e-6580-40e6-be30-2f2ca382a033\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") "
Nov 02 20:45:17.614216 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.614109    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\") pod \"7564bd9e-6580-40e6-be30-2f2ca382a033\" (UID: \"7564bd9e-6580-40e6-be30-2f2ca382a033\") "
Nov 02 20:45:17.614480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.614449    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca" pod "7564bd9e-6580-40e6-be30-2f2ca382a033" (UID: "7564bd9e-6580-40e6-be30-2f2ca382a033"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:17.616182 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.616161    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf" pod "7564bd9e-6580-40e6-be30-2f2ca382a033" (UID: "7564bd9e-6580-40e6-be30-2f2ca382a033"). InnerVolumeSpecName "kube-api-access-jr2qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:17.715586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.715496    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7564bd9e-6580-40e6-be30-2f2ca382a033-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:17.715586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:17.715526    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr2qf\" (UniqueName: \"kubernetes.io/projected/7564bd9e-6580-40e6-be30-2f2ca382a033-kube-api-access-jr2qf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:18.296805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:18.296753    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"
Nov 02 20:45:18.296805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:18.296761    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k" event={"ID":"7564bd9e-6580-40e6-be30-2f2ca382a033","Type":"ContainerDied","Data":"780be1dc49803050e9a945c74a3ad679e22aa3c116758555363d8633fa5e36b7"}
Nov 02 20:45:18.297070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:18.296840    5099 scope.go:122] "RemoveContainer" containerID="aa0cd0a70441da39d2135d6004e5a6507fd41bb67bd428ab48db2c5b95894a21"
Nov 02 20:45:18.453558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:18.453520    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:18.500734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:18.500702    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-zbx9k"]
Nov 02 20:45:19.256686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:19.256650    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" path="/var/lib/kubelet/pods/7564bd9e-6580-40e6-be30-2f2ca382a033/volumes"
Nov 02 20:45:46.094040 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:45:46.093999    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:45:55.293935 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.293897    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"]
Nov 02 20:45:55.294467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294448    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294469    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294493    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294514 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294498    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294563    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294575    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.294615 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.294583    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:45:55.297718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.297691    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.349548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.349518    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"]
Nov 02 20:45:55.394632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.394598    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:45:55.394856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.394648    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:45:55.472505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.472468    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.472683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.472547    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.472683 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.472676    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgd4k\" (UniqueName: \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.574076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.573969    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgd4k\" (UniqueName: \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.574076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.574057    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.574333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.574089    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.574755 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.574730    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.576495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.576476    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.627795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.627758    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgd4k\" (UniqueName: \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\") pod \"maas-api-revoke-sub-c1e271a9-d8ph9\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:55.910555 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:55.910525    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:56.275376 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:56.275346    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"]
Nov 02 20:45:56.276693 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:45:56.276663    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9be8f92_02fc_42bc_b285_f25f03055de4.slice/crio-2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724 WatchSource:0}: Error finding container 2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724: Status 404 returned error can't find the container with id 2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724
Nov 02 20:45:56.498712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:56.498676    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9" event={"ID":"c9be8f92-02fc-42bc-b285-f25f03055de4","Type":"ContainerStarted","Data":"2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724"}
Nov 02 20:45:57.504977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:57.504947    5099 generic.go:356] "Generic (PLEG): container finished" podID="c9be8f92-02fc-42bc-b285-f25f03055de4" containerID="93f2fc46970b06e0d3b7c5f7fa8ea8d3a533e1c43da2320439926848a32dfb1b" exitCode=0
Nov 02 20:45:57.505412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:57.504998    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9" event={"ID":"c9be8f92-02fc-42bc-b285-f25f03055de4","Type":"ContainerDied","Data":"93f2fc46970b06e0d3b7c5f7fa8ea8d3a533e1c43da2320439926848a32dfb1b"}
Nov 02 20:45:58.645717 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.645694    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:45:58.805273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.805181    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\") pod \"c9be8f92-02fc-42bc-b285-f25f03055de4\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") "
Nov 02 20:45:58.805273 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.805245    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\") pod \"c9be8f92-02fc-42bc-b285-f25f03055de4\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") "
Nov 02 20:45:58.805501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.805396    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\" (UniqueName: \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\") pod \"c9be8f92-02fc-42bc-b285-f25f03055de4\" (UID: \"c9be8f92-02fc-42bc-b285-f25f03055de4\") "
Nov 02 20:45:58.805575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.805552    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca" pod "c9be8f92-02fc-42bc-b285-f25f03055de4" (UID: "c9be8f92-02fc-42bc-b285-f25f03055de4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:45:58.805716 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.805697    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:58.807283 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.807236    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca" pod "c9be8f92-02fc-42bc-b285-f25f03055de4" (UID: "c9be8f92-02fc-42bc-b285-f25f03055de4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:45:58.807408 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.807387    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k" pod "c9be8f92-02fc-42bc-b285-f25f03055de4" (UID: "c9be8f92-02fc-42bc-b285-f25f03055de4"). InnerVolumeSpecName "kube-api-access-mgd4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:45:58.907097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.907060    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgd4k\" (UniqueName: \"kubernetes.io/projected/c9be8f92-02fc-42bc-b285-f25f03055de4-kube-api-access-mgd4k\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:58.907097 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:58.907091    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9be8f92-02fc-42bc-b285-f25f03055de4-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:45:59.519567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:59.516632    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9" event={"ID":"c9be8f92-02fc-42bc-b285-f25f03055de4","Type":"ContainerDied","Data":"2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724"}
Nov 02 20:45:59.519567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:59.516705    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2943a49abbe4e419950e855ff5e62cd07bd727dab6e34acb12118d8fb3a50724"
Nov 02 20:45:59.519567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:45:59.516820    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"
Nov 02 20:46:15.231817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.231768    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"]
Nov 02 20:46:15.232422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.232402    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:46:15.232477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.232424    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7564bd9e-6580-40e6-be30-2f2ca382a033" containerName="cleanup"
Nov 02 20:46:15.232477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.232456    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c9be8f92-02fc-42bc-b285-f25f03055de4" containerName="revoke-keys"
Nov 02 20:46:15.232477 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.232462    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9be8f92-02fc-42bc-b285-f25f03055de4" containerName="revoke-keys"
Nov 02 20:46:15.232592 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.232535    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9be8f92-02fc-42bc-b285-f25f03055de4" containerName="revoke-keys"
Nov 02 20:46:15.236012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.235993    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254295 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254266    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254359    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254389    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd74f9-e682-4a78-b870-00402c71d4f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254446    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254485    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92xj\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-kube-api-access-d92xj\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254536    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254630    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254655    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.254754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.254698    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.327838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.327795    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"]
Nov 02 20:46:15.356027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.355992    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356048    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356085    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd74f9-e682-4a78-b870-00402c71d4f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356221 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356106    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356283    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92xj\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-kube-api-access-d92xj\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356354    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356454    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356499    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.356953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.356523    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.357304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.357004    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd74f9-e682-4a78-b870-00402c71d4f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.357304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.357087    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.357304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.357131    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.357304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.357279    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.357575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.357447    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.359907 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.359885    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.360276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.360254    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.410974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.410931    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.411140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.411027    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92xj\" (UniqueName: \"kubernetes.io/projected/d1fd74f9-e682-4a78-b870-00402c71d4f0-kube-api-access-d92xj\") pod \"maas-default-gateway-openshift-default-696f47c6c-vgn28\" (UID: \"d1fd74f9-e682-4a78-b870-00402c71d4f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.548774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.548677    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:15.966146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.966118    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"]
Nov 02 20:46:15.967810 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:46:15.967756    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fd74f9_e682_4a78_b870_00402c71d4f0.slice/crio-3f879eac07227feb21c313472ede47da90d09ae74b2b0c1bfe57ea60013815b3 WatchSource:0}: Error finding container 3f879eac07227feb21c313472ede47da90d09ae74b2b0c1bfe57ea60013815b3: Status 404 returned error can't find the container with id 3f879eac07227feb21c313472ede47da90d09ae74b2b0c1bfe57ea60013815b3
Nov 02 20:46:15.970034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.969999    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:15.970145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.970078    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:15.970145 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:15.970125    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:16.094878 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:46:16.094834    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:46:16.619435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:16.619403    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" event={"ID":"d1fd74f9-e682-4a78-b870-00402c71d4f0","Type":"ContainerStarted","Data":"b0ba6e6b37cb69d17e5f4a71d049579d6cb7f2b0990a19eabd41efb259cce662"}
Nov 02 20:46:16.619827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:16.619453    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" event={"ID":"d1fd74f9-e682-4a78-b870-00402c71d4f0","Type":"ContainerStarted","Data":"3f879eac07227feb21c313472ede47da90d09ae74b2b0c1bfe57ea60013815b3"}
Nov 02 20:46:16.731388 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:16.731334    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podStartSLOduration=1.7313198079999998 podStartE2EDuration="1.731319808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:16.729760809 +0000 UTC m=+3265.604031697" watchObservedRunningTime="2026-11-02 20:46:16.731319808 +0000 UTC m=+3265.605590695"
Nov 02 20:46:17.551810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:17.551492    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:17.551810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:17.551619    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:17.551810 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:17.551662    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:18.551260 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:18.551221    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:18.551909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:18.551871    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:19.555807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:19.555249    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:19.555807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:19.555315    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:20.564809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:20.559163    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:20.564809 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:20.559228    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:21.557807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:21.557224    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:21.557807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:21.557296    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:22.549583 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:22.549540    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 20:46:22.550142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:22.549604    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 20:46:24.185997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:24.185957    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:24.186476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:24.186372    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:24.187389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:24.187367    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28"
Nov 02 20:46:28.733385 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.733350    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"]
Nov 02 20:46:28.737987 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.737964    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.787968 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.787935    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"]
Nov 02 20:46:28.802505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.802476    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mvm\" (UniqueName: \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.802682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.802527    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.802682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.802649    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.832172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.832140    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:46:28.832172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.832140    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:46:28.904229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.904199    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.904406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.904271    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mvm\" (UniqueName: \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.904406 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.904320    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.905039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.905016    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.906943 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.906919    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:28.957896 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:28.957863    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mvm\" (UniqueName: \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\") pod \"maas-api-revoke-sub-2e1f5b5f-rnd5g\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:29.051137 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:29.051038    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:29.419205 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:29.419174    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"]
Nov 02 20:46:29.420804 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:46:29.420763    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53d1deb_c928_4737_9d34_71daf35d7de3.slice/crio-9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906 WatchSource:0}: Error finding container 9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906: Status 404 returned error can't find the container with id 9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906
Nov 02 20:46:29.711773 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:29.711655    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g" event={"ID":"e53d1deb-c928-4737-9d34-71daf35d7de3","Type":"ContainerStarted","Data":"9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906"}
Nov 02 20:46:30.354426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.354393    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"]
Nov 02 20:46:30.357983 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.357964    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.418675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418646    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.418848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418678    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.418848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418698    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda36b26-75e5-45af-8fa1-ec906d88185b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.418940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418844    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.418940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418931    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.419004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418958    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.419004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.418979    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.419073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.419065    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.419110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.419095    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgns\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-kube-api-access-8lgns\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.453032 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.452997    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"]
Nov 02 20:46:30.520758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520676    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.520758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520720    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.520758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520752    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgns\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-kube-api-access-8lgns\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520813    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520833    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520853    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda36b26-75e5-45af-8fa1-ec906d88185b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.520972    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521062    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521103    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521133    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521299    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521596 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521570    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521594    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda36b26-75e5-45af-8fa1-ec906d88185b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.521667 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.521594    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.523206 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.523184    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.523431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.523413    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.574450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.574403    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.574997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.574973    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgns\" (UniqueName: \"kubernetes.io/projected/fda36b26-75e5-45af-8fa1-ec906d88185b-kube-api-access-8lgns\") pod \"maas-default-gateway-openshift-default-696f47c6c-jskmn\" (UID: \"fda36b26-75e5-45af-8fa1-ec906d88185b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.671142 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.671107    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:30.718677 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.718644    5099 generic.go:356] "Generic (PLEG): container finished" podID="e53d1deb-c928-4737-9d34-71daf35d7de3" containerID="2bfef1c985ef0e674b2c274bbcb020b4968747dde1350919a8fad23139aea19f" exitCode=0
Nov 02 20:46:30.718898 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:30.718733    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g" event={"ID":"e53d1deb-c928-4737-9d34-71daf35d7de3","Type":"ContainerDied","Data":"2bfef1c985ef0e674b2c274bbcb020b4968747dde1350919a8fad23139aea19f"}
Nov 02 20:46:31.092126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.083379    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"]
Nov 02 20:46:31.092126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.090097    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:31.092126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.090231    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:31.092126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.090315    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:46:31.727108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.727059    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" event={"ID":"fda36b26-75e5-45af-8fa1-ec906d88185b","Type":"ContainerStarted","Data":"19149fbc4fa078c24d0ce1033e73b5c1ed60a2daf50e6b3b04ec7b110709a5a3"}
Nov 02 20:46:31.727641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.727123    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" event={"ID":"fda36b26-75e5-45af-8fa1-ec906d88185b","Type":"ContainerStarted","Data":"3b2406e3f1146658e881132dcc65bd3454458972de6314bb845676496cbc54a3"}
Nov 02 20:46:31.866088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.866028    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podStartSLOduration=1.8660142880000001 podStartE2EDuration="1.866014288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:31.863952381 +0000 UTC m=+3280.738223272" watchObservedRunningTime="2026-11-02 20:46:31.866014288 +0000 UTC m=+3280.740285173"
Nov 02 20:46:31.912027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:31.912004    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:32.067619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.067512    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\") pod \"e53d1deb-c928-4737-9d34-71daf35d7de3\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") "
Nov 02 20:46:32.067619 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.067593    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\") pod \"e53d1deb-c928-4737-9d34-71daf35d7de3\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") "
Nov 02 20:46:32.067883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.067668    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\" (UniqueName: \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\") pod \"e53d1deb-c928-4737-9d34-71daf35d7de3\" (UID: \"e53d1deb-c928-4737-9d34-71daf35d7de3\") "
Nov 02 20:46:32.068087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.068061    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca" pod "e53d1deb-c928-4737-9d34-71daf35d7de3" (UID: "e53d1deb-c928-4737-9d34-71daf35d7de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:46:32.069670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.069639    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca" pod "e53d1deb-c928-4737-9d34-71daf35d7de3" (UID: "e53d1deb-c928-4737-9d34-71daf35d7de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:46:32.069873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.069849    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm" pod "e53d1deb-c928-4737-9d34-71daf35d7de3" (UID: "e53d1deb-c928-4737-9d34-71daf35d7de3"). InnerVolumeSpecName "kube-api-access-m4mvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:46:32.169539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.169488    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:32.169539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.169535    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mvm\" (UniqueName: \"kubernetes.io/projected/e53d1deb-c928-4737-9d34-71daf35d7de3-kube-api-access-m4mvm\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:32.169539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.169546    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d1deb-c928-4737-9d34-71daf35d7de3-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:32.671631 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.671600    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:32.673271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.673237    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:32.673415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.673289    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:32.733493 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.733460    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"
Nov 02 20:46:32.733974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.733491    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g" event={"ID":"e53d1deb-c928-4737-9d34-71daf35d7de3","Type":"ContainerDied","Data":"9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906"}
Nov 02 20:46:32.733974 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:32.733541    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d47afd569101df548412fae64170f0f9f0c72d4005024e8f9d19041c20ed906"
Nov 02 20:46:33.672368 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:33.672330    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:33.672568 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:33.672398    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:34.672381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:34.672339    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:34.672843 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:34.672399    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:35.672375 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:35.672340    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:35.672578 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:35.672396    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:36.672401 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:36.672365    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:36.672589 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:36.672422    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:37.671609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:37.671573    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 20:46:37.671815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:37.671628    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 20:46:39.374471 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:39.374435    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:39.374977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:39.374733    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:39.375573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:39.375555    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn"
Nov 02 20:46:41.113647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.113618    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"]
Nov 02 20:46:41.114121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.114070    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e53d1deb-c928-4737-9d34-71daf35d7de3" containerName="revoke-keys"
Nov 02 20:46:41.114121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.114083    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53d1deb-c928-4737-9d34-71daf35d7de3" containerName="revoke-keys"
Nov 02 20:46:41.114227 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.114174    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53d1deb-c928-4737-9d34-71daf35d7de3" containerName="revoke-keys"
Nov 02 20:46:41.117611 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.117594    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.162228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.162192    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6c7x\" (UniqueName: \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.162427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.162238    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.162427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.162362    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.169384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.169351    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"]
Nov 02 20:46:41.214122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.214094    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:46:41.214122 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.214104    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:46:41.263549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.263520    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.263729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.263593    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6c7x\" (UniqueName: \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.263729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.263625    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.264346 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.264321    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.265903 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.265877    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.320071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.320040    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6c7x\" (UniqueName: \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\") pod \"maas-api-revoke-sub-ec843018-pj5js\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.430688 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.430603    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:41.795422 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.795390    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"]
Nov 02 20:46:41.797116 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:46:41.797080    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edf0467_6f8a_471f_8158_9b9434f6902a.slice/crio-efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5 WatchSource:0}: Error finding container efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5: Status 404 returned error can't find the container with id efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5
Nov 02 20:46:41.811966 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:41.811930    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js" event={"ID":"1edf0467-6f8a-471f-8158-9b9434f6902a","Type":"ContainerStarted","Data":"efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5"}
Nov 02 20:46:42.822758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:42.822644    5099 generic.go:356] "Generic (PLEG): container finished" podID="1edf0467-6f8a-471f-8158-9b9434f6902a" containerID="418a244855194ec81055ae145d7db6fe0a00a2d72d6a135a153e210bb47bb744" exitCode=0
Nov 02 20:46:42.823314 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:42.822743    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js" event={"ID":"1edf0467-6f8a-471f-8158-9b9434f6902a","Type":"ContainerDied","Data":"418a244855194ec81055ae145d7db6fe0a00a2d72d6a135a153e210bb47bb744"}
Nov 02 20:46:44.031582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.031549    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:44.092531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.092440    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\" (UniqueName: \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\") pod \"1edf0467-6f8a-471f-8158-9b9434f6902a\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") "
Nov 02 20:46:44.092531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.092499    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\") pod \"1edf0467-6f8a-471f-8158-9b9434f6902a\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") "
Nov 02 20:46:44.092924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.092560    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\") pod \"1edf0467-6f8a-471f-8158-9b9434f6902a\" (UID: \"1edf0467-6f8a-471f-8158-9b9434f6902a\") "
Nov 02 20:46:44.093356 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.093329    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca" pod "1edf0467-6f8a-471f-8158-9b9434f6902a" (UID: "1edf0467-6f8a-471f-8158-9b9434f6902a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:46:44.096979 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.096952    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca" pod "1edf0467-6f8a-471f-8158-9b9434f6902a" (UID: "1edf0467-6f8a-471f-8158-9b9434f6902a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:46:44.097708 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.097680    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x" pod "1edf0467-6f8a-471f-8158-9b9434f6902a" (UID: "1edf0467-6f8a-471f-8158-9b9434f6902a"). InnerVolumeSpecName "kube-api-access-h6c7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:46:44.208661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.207417    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6c7x\" (UniqueName: \"kubernetes.io/projected/1edf0467-6f8a-471f-8158-9b9434f6902a-kube-api-access-h6c7x\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:44.208661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.207460    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:44.208661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.207477    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edf0467-6f8a-471f-8158-9b9434f6902a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:46:44.834607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.834572    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"
Nov 02 20:46:44.834607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.834591    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js" event={"ID":"1edf0467-6f8a-471f-8158-9b9434f6902a","Type":"ContainerDied","Data":"efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5"}
Nov 02 20:46:44.834875 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:44.834639    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc9a058ee60298e6e8586a70596a241465cbc55e53f3a6cd988a7db665477b5"
Nov 02 20:46:46.096363 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:46:46.096318    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:46:52.011586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:52.011559    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:46:52.060871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:46:52.060843    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:47:00.584808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.584184    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"]
Nov 02 20:47:00.585354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.584892    5099 cpu_manager.go:414] "containerMap: removing container" podUID="1edf0467-6f8a-471f-8158-9b9434f6902a" containerName="revoke-keys"
Nov 02 20:47:00.585354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.584916    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edf0467-6f8a-471f-8158-9b9434f6902a" containerName="revoke-keys"
Nov 02 20:47:00.585354 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.585073    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edf0467-6f8a-471f-8158-9b9434f6902a" containerName="revoke-keys"
Nov 02 20:47:00.593642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.592973    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.716520 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.716478    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.720820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.720693    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.721119 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.721093    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.721523 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.721502    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt9z\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-kube-api-access-tvt9z\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.721725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.721707    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.721924 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.721906    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.722617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.722594    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.722899 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.722857    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.723143 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.723124    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.724431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.724408    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"]
Nov 02 20:47:00.824280 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824247    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt9z\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-kube-api-access-tvt9z\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824492    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824512 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824514    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824540    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824587    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824633    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824667    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824695    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.824725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.824718    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.825524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.825337    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.825524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.825390    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.825524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.825483    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.825524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.825497    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.825806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.825564    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.827034 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.826996    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.827499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.827480    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.910815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.910240    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.913508 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.913445    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt9z\" (UniqueName: \"kubernetes.io/projected/fb1fe38d-88dd-4f15-8f9a-8613c5161003-kube-api-access-tvt9z\") pod \"maas-default-gateway-openshift-default-696f47c6c-p27sn\" (UID: \"fb1fe38d-88dd-4f15-8f9a-8613c5161003\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:00.932225 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:00.932198    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:01.489146 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.489108    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"]
Nov 02 20:47:01.490261 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:47:01.490224    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1fe38d_88dd_4f15_8f9a_8613c5161003.slice/crio-b5bc5abee895295ddbdd81f6d0e5086639c07340e94aa55e9bf207d138a9a824 WatchSource:0}: Error finding container b5bc5abee895295ddbdd81f6d0e5086639c07340e94aa55e9bf207d138a9a824: Status 404 returned error can't find the container with id b5bc5abee895295ddbdd81f6d0e5086639c07340e94aa55e9bf207d138a9a824
Nov 02 20:47:01.492684 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.492648    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:47:01.492817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.492726    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:47:01.492817 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.492769    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 20:47:01.935444 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.935404    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" event={"ID":"fb1fe38d-88dd-4f15-8f9a-8613c5161003","Type":"ContainerStarted","Data":"56907075544ac9c64dd4899ade8ef63edf4cc933791ad2ccd9cbe2d8888dc37b"}
Nov 02 20:47:01.935862 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:01.935470    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" event={"ID":"fb1fe38d-88dd-4f15-8f9a-8613c5161003","Type":"ContainerStarted","Data":"b5bc5abee895295ddbdd81f6d0e5086639c07340e94aa55e9bf207d138a9a824"}
Nov 02 20:47:02.053806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:02.051189    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podStartSLOduration=2.051171525 podStartE2EDuration="2.051171525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:02.049599536 +0000 UTC m=+3310.923870529" watchObservedRunningTime="2026-11-02 20:47:02.051171525 +0000 UTC m=+3310.925442414"
Nov 02 20:47:02.932505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:02.932465    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:02.934202 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:02.934174    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:02.934335 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:02.934233    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:03.932838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:03.932771    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:03.933238 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:03.932860    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:04.933120 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:04.933082    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:04.933535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:04.933137    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:05.933018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:05.932981    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:05.933209 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:05.933033    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:06.933090 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:06.933058    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:06.933303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:06.933110    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:07.933427 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:07.933387    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:07.933863 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:07.933450    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:08.932634 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:08.932594    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-p27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 20:47:08.932897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:08.932656    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn" podUID="fb1fe38d-88dd-4f15-8f9a-8613c5161003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 20:47:09.936674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:09.936644    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:09.937193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:09.937083    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:09.937740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:09.937724    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-p27sn"
Nov 02 20:47:16.092981 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:47:16.092936    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:47:26.011937 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.011903    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"]
Nov 02 20:47:26.015830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.015809    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.068259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.068229    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"]
Nov 02 20:47:26.083195 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.083156    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5vj\" (UniqueName: \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.083360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.083214    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.083360 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.083335    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.112499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.112470    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:47:26.112671 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.112599    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:47:26.184150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.184116    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5vj\" (UniqueName: \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.184332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.184170    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.184332 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.184225    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.184918 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.184892    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.186661 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.186642    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.238313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.238279    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5vj\" (UniqueName: \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\") pod \"maas-api-revoke-sub-046e1343-dpxpk\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.329301 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.329217    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:26.692250 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:26.692219    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"]
Nov 02 20:47:26.693374 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:47:26.693346    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b69a301_6e8b_43d2_b702_7722a769ad50.slice/crio-8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163 WatchSource:0}: Error finding container 8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163: Status 404 returned error can't find the container with id 8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163
Nov 02 20:47:27.097080 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:27.097041    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk" event={"ID":"0b69a301-6e8b-43d2-b702-7722a769ad50","Type":"ContainerStarted","Data":"c850468b4618400f595e84701a725e40f5058df776f4be23acb31f22fcd9dd43"}
Nov 02 20:47:27.097506 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:27.097129    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk" event={"ID":"0b69a301-6e8b-43d2-b702-7722a769ad50","Type":"ContainerStarted","Data":"8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163"}
Nov 02 20:47:27.206838 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:27.206729    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk" podStartSLOduration=2.206715086 podStartE2EDuration="2.206715086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:27.203400179 +0000 UTC m=+3336.077671067" watchObservedRunningTime="2026-11-02 20:47:27.206715086 +0000 UTC m=+3336.080985973"
Nov 02 20:47:28.104473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:28.104437    5099 generic.go:356] "Generic (PLEG): container finished" podID="0b69a301-6e8b-43d2-b702-7722a769ad50" containerID="c850468b4618400f595e84701a725e40f5058df776f4be23acb31f22fcd9dd43" exitCode=0
Nov 02 20:47:28.104905 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:28.104521    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk" event={"ID":"0b69a301-6e8b-43d2-b702-7722a769ad50","Type":"ContainerDied","Data":"c850468b4618400f595e84701a725e40f5058df776f4be23acb31f22fcd9dd43"}
Nov 02 20:47:29.248738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.248710    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:29.315798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.315758    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\") pod \"0b69a301-6e8b-43d2-b702-7722a769ad50\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") "
Nov 02 20:47:29.316005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.315906    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\" (UniqueName: \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\") pod \"0b69a301-6e8b-43d2-b702-7722a769ad50\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") "
Nov 02 20:47:29.316005 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.315978    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\") pod \"0b69a301-6e8b-43d2-b702-7722a769ad50\" (UID: \"0b69a301-6e8b-43d2-b702-7722a769ad50\") "
Nov 02 20:47:29.316183 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.316150    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca" pod "0b69a301-6e8b-43d2-b702-7722a769ad50" (UID: "0b69a301-6e8b-43d2-b702-7722a769ad50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:47:29.316342 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.316324    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:29.318100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.318079    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj" pod "0b69a301-6e8b-43d2-b702-7722a769ad50" (UID: "0b69a301-6e8b-43d2-b702-7722a769ad50"). InnerVolumeSpecName "kube-api-access-sl5vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:47:29.318100 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.318084    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca" pod "0b69a301-6e8b-43d2-b702-7722a769ad50" (UID: "0b69a301-6e8b-43d2-b702-7722a769ad50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:47:29.417208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.417120    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5vj\" (UniqueName: \"kubernetes.io/projected/0b69a301-6e8b-43d2-b702-7722a769ad50-kube-api-access-sl5vj\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:29.417208 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:29.417151    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a301-6e8b-43d2-b702-7722a769ad50-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:30.116069 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:30.116031    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk" event={"ID":"0b69a301-6e8b-43d2-b702-7722a769ad50","Type":"ContainerDied","Data":"8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163"}
Nov 02 20:47:30.116226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:30.116085    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e32acc3905f7c0f1dbf5f8738a84466bc090309066320b746eeeb1ee9c78163"
Nov 02 20:47:30.116226 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:30.116041    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"
Nov 02 20:47:36.770488 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.770448    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"]
Nov 02 20:47:36.771363 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.771338    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0b69a301-6e8b-43d2-b702-7722a769ad50" containerName="revoke-keys"
Nov 02 20:47:36.771413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.771370    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b69a301-6e8b-43d2-b702-7722a769ad50" containerName="revoke-keys"
Nov 02 20:47:36.771572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.771556    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b69a301-6e8b-43d2-b702-7722a769ad50" containerName="revoke-keys"
Nov 02 20:47:36.775827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.775802    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.826336 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.826302    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"]
Nov 02 20:47:36.870248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.870216    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:47:36.870248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.870217    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:47:36.893115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.893082    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.893277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.893165    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cggg\" (UniqueName: \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.893277 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.893198    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.994481 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.994443    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.994638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.994508    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cggg\" (UniqueName: \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.994638 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.994541    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.995212 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.995188    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:36.996877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:36.996859    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:37.048887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:37.048814    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cggg\" (UniqueName: \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\") pod \"maas-api-revoke-sub-d3fba056-qg64t\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:37.089307 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:37.089277    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:37.451956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:37.451928    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"]
Nov 02 20:47:37.455116 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:47:37.455073    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6a7fcd_8645_41ab_a435_5a907b84e0c8.slice/crio-f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20 WatchSource:0}: Error finding container f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20: Status 404 returned error can't find the container with id f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20
Nov 02 20:47:38.163181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:38.163142    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t" event={"ID":"ee6a7fcd-8645-41ab-a435-5a907b84e0c8","Type":"ContainerStarted","Data":"38e2bea41b58754a7c0cb3b1b14684ef324475f4dd4a9cde63ed284f4f186c8e"}
Nov 02 20:47:38.163593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:38.163204    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t" event={"ID":"ee6a7fcd-8645-41ab-a435-5a907b84e0c8","Type":"ContainerStarted","Data":"f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20"}
Nov 02 20:47:38.273074 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:38.273024    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t" podStartSLOduration=2.273010517 podStartE2EDuration="2.273010517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:38.269984673 +0000 UTC m=+3347.144255563" watchObservedRunningTime="2026-11-02 20:47:38.273010517 +0000 UTC m=+3347.147281404"
Nov 02 20:47:39.206013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:39.205965    5099 generic.go:356] "Generic (PLEG): container finished" podID="ee6a7fcd-8645-41ab-a435-5a907b84e0c8" containerID="38e2bea41b58754a7c0cb3b1b14684ef324475f4dd4a9cde63ed284f4f186c8e" exitCode=0
Nov 02 20:47:39.206509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:39.206205    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t" event={"ID":"ee6a7fcd-8645-41ab-a435-5a907b84e0c8","Type":"ContainerDied","Data":"38e2bea41b58754a7c0cb3b1b14684ef324475f4dd4a9cde63ed284f4f186c8e"}
Nov 02 20:47:41.260806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.257283    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:41.309861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.308903    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t" event={"ID":"ee6a7fcd-8645-41ab-a435-5a907b84e0c8","Type":"ContainerDied","Data":"f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20"}
Nov 02 20:47:41.309861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.308978    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8bd83a4b0069cecce18e6e30ab03b880f7d043baa142729cde981c30d9a5e20"
Nov 02 20:47:41.309861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.309338    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\" (UniqueName: \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\") pod \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") "
Nov 02 20:47:41.309861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.309418    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\") pod \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") "
Nov 02 20:47:41.309861 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.309583    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\") pod \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\" (UID: \"ee6a7fcd-8645-41ab-a435-5a907b84e0c8\") "
Nov 02 20:47:41.314805 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.311133    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca" pod "ee6a7fcd-8645-41ab-a435-5a907b84e0c8" (UID: "ee6a7fcd-8645-41ab-a435-5a907b84e0c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:47:41.333039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.333005    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg" pod "ee6a7fcd-8645-41ab-a435-5a907b84e0c8" (UID: "ee6a7fcd-8645-41ab-a435-5a907b84e0c8"). InnerVolumeSpecName "kube-api-access-4cggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:47:41.344947 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.344896    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca" pod "ee6a7fcd-8645-41ab-a435-5a907b84e0c8" (UID: "ee6a7fcd-8645-41ab-a435-5a907b84e0c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:47:41.418564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.418474    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:41.418564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.418517    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cggg\" (UniqueName: \"kubernetes.io/projected/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-kube-api-access-4cggg\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:41.418564 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:41.418533    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a7fcd-8645-41ab-a435-5a907b84e0c8-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:47:42.278949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:47:42.278914    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"
Nov 02 20:47:46.094390 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:47:46.094351    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:48:11.458672 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.458580    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"]
Nov 02 20:48:11.459326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.459303    5099 cpu_manager.go:414] "containerMap: removing container" podUID="ee6a7fcd-8645-41ab-a435-5a907b84e0c8" containerName="revoke-keys"
Nov 02 20:48:11.459432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.459328    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6a7fcd-8645-41ab-a435-5a907b84e0c8" containerName="revoke-keys"
Nov 02 20:48:11.459509 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.459496    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6a7fcd-8645-41ab-a435-5a907b84e0c8" containerName="revoke-keys"
Nov 02 20:48:11.463099 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.463074    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.514056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.514018    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"]
Nov 02 20:48:11.534575 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.534545    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.534744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.534585    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqrk\" (UniqueName: \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.534744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.534617    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.558266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.558231    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:48:11.558266 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.558253    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:48:11.635675 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.635637    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.635889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.635684    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqrk\" (UniqueName: \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.635889 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.635711    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.636333 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.636314    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.638185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.638156    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.690827 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.690768    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqrk\" (UniqueName: \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\") pod \"maas-api-revoke-sub-ae594979-2r4l2\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:11.776215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:11.776140    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:12.142607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:12.142578    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"]
Nov 02 20:48:12.143992 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:48:12.143964    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dd6492_3b74_448a_91aa_c13ca5cd8601.slice/crio-93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d WatchSource:0}: Error finding container 93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d: Status 404 returned error can't find the container with id 93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d
Nov 02 20:48:12.501377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:12.501340    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2" event={"ID":"c7dd6492-3b74-448a-91aa-c13ca5cd8601","Type":"ContainerStarted","Data":"93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d"}
Nov 02 20:48:13.508964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:13.508928    5099 generic.go:356] "Generic (PLEG): container finished" podID="c7dd6492-3b74-448a-91aa-c13ca5cd8601" containerID="aa1329192f9035c4cb5d670d55f11fdb84571c174b860dcb794d03df0f59dd15" exitCode=0
Nov 02 20:48:13.509434 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:13.508980    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2" event={"ID":"c7dd6492-3b74-448a-91aa-c13ca5cd8601","Type":"ContainerDied","Data":"aa1329192f9035c4cb5d670d55f11fdb84571c174b860dcb794d03df0f59dd15"}
Nov 02 20:48:14.658383 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.658357    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:14.770977 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.770888    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\" (UniqueName: \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\") pod \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") "
Nov 02 20:48:14.771161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.770985    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\") pod \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") "
Nov 02 20:48:14.771161 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.771064    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\") pod \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\" (UID: \"c7dd6492-3b74-448a-91aa-c13ca5cd8601\") "
Nov 02 20:48:14.771740 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.771705    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca" pod "c7dd6492-3b74-448a-91aa-c13ca5cd8601" (UID: "c7dd6492-3b74-448a-91aa-c13ca5cd8601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:48:14.773746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.773723    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk" pod "c7dd6492-3b74-448a-91aa-c13ca5cd8601" (UID: "c7dd6492-3b74-448a-91aa-c13ca5cd8601"). InnerVolumeSpecName "kube-api-access-hrqrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:48:14.773746 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.773734    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca" pod "c7dd6492-3b74-448a-91aa-c13ca5cd8601" (UID: "c7dd6492-3b74-448a-91aa-c13ca5cd8601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:48:14.872051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.872020    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqrk\" (UniqueName: \"kubernetes.io/projected/c7dd6492-3b74-448a-91aa-c13ca5cd8601-kube-api-access-hrqrk\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:14.872051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.872048    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:14.872051 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:14.872057    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dd6492-3b74-448a-91aa-c13ca5cd8601-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:15.520763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:15.520675    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2" event={"ID":"c7dd6492-3b74-448a-91aa-c13ca5cd8601","Type":"ContainerDied","Data":"93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d"}
Nov 02 20:48:15.520763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:15.520734    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93efbc2dad386a09b285a51ae4df4bdb956dca5727d9d508730451658835554d"
Nov 02 20:48:15.520763 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:15.520691    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"
Nov 02 20:48:16.093045 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:48:16.093004    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:48:39.368417 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.368383    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"]
Nov 02 20:48:39.369033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.369003    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c7dd6492-3b74-448a-91aa-c13ca5cd8601" containerName="revoke-keys"
Nov 02 20:48:39.369033 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.369030    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7dd6492-3b74-448a-91aa-c13ca5cd8601" containerName="revoke-keys"
Nov 02 20:48:39.369204 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.369194    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7dd6492-3b74-448a-91aa-c13ca5cd8601" containerName="revoke-keys"
Nov 02 20:48:39.372829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.372810    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.423841 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.423800    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"]
Nov 02 20:48:39.467655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.467623    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:48:39.467871 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.467683    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:48:39.478595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.478258    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.478595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.478446    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjcb\" (UniqueName: \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.478595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.478558    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.579725 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.579688    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.579919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.579766    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.579919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.579854    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjcb\" (UniqueName: \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.580543 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.580509    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.582113 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.582088    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.634257 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.634179    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjcb\" (UniqueName: \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\") pod \"maas-api-revoke-sub-a4aef7d7-cpbj8\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:39.691098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:39.691067    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:40.054516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:40.054488    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"]
Nov 02 20:48:40.057070 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:48:40.057042    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05b67f3_76f6_4c52_810f_b566a579399b.slice/crio-ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be WatchSource:0}: Error finding container ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be: Status 404 returned error can't find the container with id ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be
Nov 02 20:48:40.696296 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:40.696258    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8" event={"ID":"e05b67f3-76f6-4c52-810f-b566a579399b","Type":"ContainerStarted","Data":"fd3762abb527b6f15fad76d7a92c9a7c1be40fbf2c2bd91983e2236375862738"}
Nov 02 20:48:40.696697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:40.696324    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8" event={"ID":"e05b67f3-76f6-4c52-810f-b566a579399b","Type":"ContainerStarted","Data":"ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be"}
Nov 02 20:48:40.806347 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:40.806284    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8" podStartSLOduration=1.806265388 podStartE2EDuration="1.806265388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:40.803219128 +0000 UTC m=+3409.677490017" watchObservedRunningTime="2026-11-02 20:48:40.806265388 +0000 UTC m=+3409.680536279"
Nov 02 20:48:41.042248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:41.042207    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"]
Nov 02 20:48:41.044848 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:41.044822    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2503c-jb7sw"]
Nov 02 20:48:41.257115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:41.257042    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eec3bf9-86d8-4276-a6d7-41a00076011b" path="/var/lib/kubelet/pods/7eec3bf9-86d8-4276-a6d7-41a00076011b/volumes"
Nov 02 20:48:41.703490 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:41.703453    5099 generic.go:356] "Generic (PLEG): container finished" podID="e05b67f3-76f6-4c52-810f-b566a579399b" containerID="fd3762abb527b6f15fad76d7a92c9a7c1be40fbf2c2bd91983e2236375862738" exitCode=0
Nov 02 20:48:41.703926 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:41.703538    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8" event={"ID":"e05b67f3-76f6-4c52-810f-b566a579399b","Type":"ContainerDied","Data":"fd3762abb527b6f15fad76d7a92c9a7c1be40fbf2c2bd91983e2236375862738"}
Nov 02 20:48:42.850421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:42.850398    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:43.013680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.013601    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\") pod \"e05b67f3-76f6-4c52-810f-b566a579399b\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") "
Nov 02 20:48:43.013680 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.013637    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\") pod \"e05b67f3-76f6-4c52-810f-b566a579399b\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") "
Nov 02 20:48:43.013904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.013710    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\" (UniqueName: \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\") pod \"e05b67f3-76f6-4c52-810f-b566a579399b\" (UID: \"e05b67f3-76f6-4c52-810f-b566a579399b\") "
Nov 02 20:48:43.014007 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.013984    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca" pod "e05b67f3-76f6-4c52-810f-b566a579399b" (UID: "e05b67f3-76f6-4c52-810f-b566a579399b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:48:43.015829 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.015809    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb" pod "e05b67f3-76f6-4c52-810f-b566a579399b" (UID: "e05b67f3-76f6-4c52-810f-b566a579399b"). InnerVolumeSpecName "kube-api-access-bpjcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:48:43.015941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.015927    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca" pod "e05b67f3-76f6-4c52-810f-b566a579399b" (UID: "e05b67f3-76f6-4c52-810f-b566a579399b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:48:43.115476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.115443    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:43.115476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.115475    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05b67f3-76f6-4c52-810f-b566a579399b-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:43.115691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.115486    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjcb\" (UniqueName: \"kubernetes.io/projected/e05b67f3-76f6-4c52-810f-b566a579399b-kube-api-access-bpjcb\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:48:43.715949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.715919    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"
Nov 02 20:48:43.716127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.715969    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8" event={"ID":"e05b67f3-76f6-4c52-810f-b566a579399b","Type":"ContainerDied","Data":"ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be"}
Nov 02 20:48:43.716127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:48:43.716020    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba3793f807314d0f7d81f8455cd31844582fd31fa4d504999bf762be5cb914be"
Nov 02 20:48:46.092928 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:48:46.092888    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:49:00.068577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:00.068524    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"]
Nov 02 20:49:00.114521 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:00.114480    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6687192-64cjn"]
Nov 02 20:49:01.306030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:01.305550    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44cff28-ea8d-48bf-b65d-ef21388f66a3" path="/var/lib/kubelet/pods/c44cff28-ea8d-48bf-b65d-ef21388f66a3/volumes"
Nov 02 20:49:12.932274 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.932238    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"]
Nov 02 20:49:12.932748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.932733    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e05b67f3-76f6-4c52-810f-b566a579399b" containerName="revoke-keys"
Nov 02 20:49:12.932748 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.932748    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05b67f3-76f6-4c52-810f-b566a579399b" containerName="revoke-keys"
Nov 02 20:49:12.932877 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.932868    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05b67f3-76f6-4c52-810f-b566a579399b" containerName="revoke-keys"
Nov 02 20:49:12.936412 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.936392    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:12.988624 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:12.988594    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"]
Nov 02 20:49:13.033269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.033242    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:49:13.033487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.033260    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:49:13.117495 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.117462    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.117664 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.117510    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.117712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.117670    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2499v\" (UniqueName: \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.219123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.219040    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2499v\" (UniqueName: \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.219123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.219108    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.219321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.219160    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.219882 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.219862    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.221548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.221524    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.273440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.273404    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2499v\" (UniqueName: \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\") pod \"maas-api-revoke-sub-79805ca9-2srk2\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.549438 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.549347    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:13.915582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.915556    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"]
Nov 02 20:49:13.916026 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:49:13.915998    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f41515_4fd4_4c62_a50f_72fb73ad62b1.slice/crio-61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8 WatchSource:0}: Error finding container 61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8: Status 404 returned error can't find the container with id 61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8
Nov 02 20:49:13.958192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:13.958159    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2" event={"ID":"42f41515-4fd4-4c62-a50f-72fb73ad62b1","Type":"ContainerStarted","Data":"61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8"}
Nov 02 20:49:14.965058 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:14.965019    5099 generic.go:356] "Generic (PLEG): container finished" podID="42f41515-4fd4-4c62-a50f-72fb73ad62b1" containerID="bed6293f5af4b593913b419abba3c282641f410a009fc24922f5f18120781ba6" exitCode=0
Nov 02 20:49:14.965473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:14.965082    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2" event={"ID":"42f41515-4fd4-4c62-a50f-72fb73ad62b1","Type":"ContainerDied","Data":"bed6293f5af4b593913b419abba3c282641f410a009fc24922f5f18120781ba6"}
Nov 02 20:49:16.099342 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:49:16.099304    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:49:16.116645 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.116622    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:16.145722 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.145687    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\") pod \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") "
Nov 02 20:49:16.145940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.145732    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\") pod \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") "
Nov 02 20:49:16.145940 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.145864    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\" (UniqueName: \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\") pod \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\" (UID: \"42f41515-4fd4-4c62-a50f-72fb73ad62b1\") "
Nov 02 20:49:16.146116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.146091    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca" pod "42f41515-4fd4-4c62-a50f-72fb73ad62b1" (UID: "42f41515-4fd4-4c62-a50f-72fb73ad62b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:49:16.146286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.146263    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:16.147919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.147897    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v" pod "42f41515-4fd4-4c62-a50f-72fb73ad62b1" (UID: "42f41515-4fd4-4c62-a50f-72fb73ad62b1"). InnerVolumeSpecName "kube-api-access-2499v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:49:16.147997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.147976    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca" pod "42f41515-4fd4-4c62-a50f-72fb73ad62b1" (UID: "42f41515-4fd4-4c62-a50f-72fb73ad62b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:49:16.247269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.247183    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2499v\" (UniqueName: \"kubernetes.io/projected/42f41515-4fd4-4c62-a50f-72fb73ad62b1-kube-api-access-2499v\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:16.247269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.247216    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f41515-4fd4-4c62-a50f-72fb73ad62b1-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:16.977305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.977261    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2" event={"ID":"42f41515-4fd4-4c62-a50f-72fb73ad62b1","Type":"ContainerDied","Data":"61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8"}
Nov 02 20:49:16.977305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.977284    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"
Nov 02 20:49:16.977305 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:16.977314    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ac806a29ce72729346901d0c7614d28e078fe9ba1839cec0f85b8783f67af8"
Nov 02 20:49:36.619269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.619180    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"]
Nov 02 20:49:36.619864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.619636    5099 cpu_manager.go:414] "containerMap: removing container" podUID="42f41515-4fd4-4c62-a50f-72fb73ad62b1" containerName="revoke-keys"
Nov 02 20:49:36.619864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.619647    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f41515-4fd4-4c62-a50f-72fb73ad62b1" containerName="revoke-keys"
Nov 02 20:49:36.619864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.619724    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f41515-4fd4-4c62-a50f-72fb73ad62b1" containerName="revoke-keys"
Nov 02 20:49:36.623253 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.623225    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.712115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.712076    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.712312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.712130    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrfd\" (UniqueName: \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.712312 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.712293    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.720892 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.720866    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:49:36.721127 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.721104    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:49:36.723774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.723755    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"]
Nov 02 20:49:36.813136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.813102    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrfd\" (UniqueName: \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.813303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.813166    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.813303 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.813271    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.813904 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.813880    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.815585 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.815565    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.867101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.867069    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrfd\" (UniqueName: \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\") pod \"maas-api-revoke-sub-0b7e8edb-hl68c\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:36.936379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:36.936289    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:37.299254 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:37.299224    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"]
Nov 02 20:49:37.299710 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:49:37.299686    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0442ea78_76ac_4032_a463_68194a68e950.slice/crio-181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6 WatchSource:0}: Error finding container 181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6: Status 404 returned error can't find the container with id 181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6
Nov 02 20:49:38.192643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:38.192610    5099 generic.go:356] "Generic (PLEG): container finished" podID="0442ea78-76ac-4032-a463-68194a68e950" containerID="406ee60144ab5e855fbccc91e477c2956f2ce5508af17e672141b32160090d4b" exitCode=0
Nov 02 20:49:38.193042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:38.192669    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c" event={"ID":"0442ea78-76ac-4032-a463-68194a68e950","Type":"ContainerDied","Data":"406ee60144ab5e855fbccc91e477c2956f2ce5508af17e672141b32160090d4b"}
Nov 02 20:49:38.193042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:38.192707    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c" event={"ID":"0442ea78-76ac-4032-a463-68194a68e950","Type":"ContainerStarted","Data":"181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6"}
Nov 02 20:49:39.334679 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.334654    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:39.442400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.442362    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\" (UniqueName: \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\") pod \"0442ea78-76ac-4032-a463-68194a68e950\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") "
Nov 02 20:49:39.442652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.442427    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\") pod \"0442ea78-76ac-4032-a463-68194a68e950\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") "
Nov 02 20:49:39.442652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.442533    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\") pod \"0442ea78-76ac-4032-a463-68194a68e950\" (UID: \"0442ea78-76ac-4032-a463-68194a68e950\") "
Nov 02 20:49:39.443455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.443402    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca" pod "0442ea78-76ac-4032-a463-68194a68e950" (UID: "0442ea78-76ac-4032-a463-68194a68e950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:49:39.445649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.445607    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca" pod "0442ea78-76ac-4032-a463-68194a68e950" (UID: "0442ea78-76ac-4032-a463-68194a68e950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:49:39.445649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.445609    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd" pod "0442ea78-76ac-4032-a463-68194a68e950" (UID: "0442ea78-76ac-4032-a463-68194a68e950"). InnerVolumeSpecName "kube-api-access-4jrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:49:39.543573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.543541    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0442ea78-76ac-4032-a463-68194a68e950-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:39.543573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.543568    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrfd\" (UniqueName: \"kubernetes.io/projected/0442ea78-76ac-4032-a463-68194a68e950-kube-api-access-4jrfd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:39.543573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:39.543579    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0442ea78-76ac-4032-a463-68194a68e950-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:49:40.207545 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:40.207507    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"
Nov 02 20:49:40.207718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:40.207539    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c" event={"ID":"0442ea78-76ac-4032-a463-68194a68e950","Type":"ContainerDied","Data":"181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6"}
Nov 02 20:49:40.207718 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:40.207588    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="181d511a0c0762b0a5656f639ca31c4c5631feebd9c7134937e8e31ca32c03d6"
Nov 02 20:49:46.093755 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:49:46.093716    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:49:50.058807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:50.056052    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"]
Nov 02 20:49:50.076026 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:50.075990    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m967m"]
Nov 02 20:49:51.434138 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:51.434095    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a09819-2406-4f54-806c-f312acf2b850" path="/var/lib/kubelet/pods/f5a09819-2406-4f54-806c-f312acf2b850/volumes"
Nov 02 20:49:56.444837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:56.444799    5099 scope.go:122] "RemoveContainer" containerID="4320c3ae920fdcce82b0cf5eecd88ff5ea0e78875f9394dfd28a78d13db00cff"
Nov 02 20:49:56.454507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:56.454487    5099 scope.go:122] "RemoveContainer" containerID="f734cf12fed4181e1729b766b57815fd1cfd70c52bec1b03dca3ed10d169b589"
Nov 02 20:49:56.463155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:56.463135    5099 scope.go:122] "RemoveContainer" containerID="da1cccebb5b1b5b4f516cc613fb5b98fe4eeb38e5aad682bab24f1a7dee58cec"
Nov 02 20:49:57.832326 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.832289    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"]
Nov 02 20:49:57.832758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.832733    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0442ea78-76ac-4032-a463-68194a68e950" containerName="revoke-keys"
Nov 02 20:49:57.832758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.832744    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0442ea78-76ac-4032-a463-68194a68e950" containerName="revoke-keys"
Nov 02 20:49:57.832869 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.832854    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0442ea78-76ac-4032-a463-68194a68e950" containerName="revoke-keys"
Nov 02 20:49:57.839313 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.839281    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:57.887453 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.887419    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"]
Nov 02 20:49:57.931826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.931759    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:49:57.931826 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.931802    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:49:57.942338 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.942313    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ng5\" (UniqueName: \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:57.942452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.942431    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:57.942491 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:57.942472    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.043704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.043671    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.043704 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.043710    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.043957 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.043760    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ng5\" (UniqueName: \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.044381 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.044359    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.046104 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.046083    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.098155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.098080    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ng5\" (UniqueName: \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\") pod \"maas-api-revoke-sub-3d4156f1-9sm89\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.153909 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.153869    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:49:58.520361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:58.520328    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"]
Nov 02 20:49:58.521190 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:49:58.521162    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf35544_c072_4fe6_852c_7dee622564b2.slice/crio-be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e WatchSource:0}: Error finding container be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e: Status 404 returned error can't find the container with id be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e
Nov 02 20:49:59.479035 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:59.479003    5099 generic.go:356] "Generic (PLEG): container finished" podID="bcf35544-c072-4fe6-852c-7dee622564b2" containerID="82636e36f26659196604dea611105368f65e5eb8a17afe2532d59db9024b78a4" exitCode=0
Nov 02 20:49:59.479435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:59.479080    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89" event={"ID":"bcf35544-c072-4fe6-852c-7dee622564b2","Type":"ContainerDied","Data":"82636e36f26659196604dea611105368f65e5eb8a17afe2532d59db9024b78a4"}
Nov 02 20:49:59.479435 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:49:59.479136    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89" event={"ID":"bcf35544-c072-4fe6-852c-7dee622564b2","Type":"ContainerStarted","Data":"be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e"}
Nov 02 20:50:00.626833 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.626773    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:50:00.772982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.772892    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\") pod \"bcf35544-c072-4fe6-852c-7dee622564b2\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") "
Nov 02 20:50:00.772982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.772959    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\") pod \"bcf35544-c072-4fe6-852c-7dee622564b2\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") "
Nov 02 20:50:00.773172 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.772988    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\" (UniqueName: \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\") pod \"bcf35544-c072-4fe6-852c-7dee622564b2\" (UID: \"bcf35544-c072-4fe6-852c-7dee622564b2\") "
Nov 02 20:50:00.773370 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.773344    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca" pod "bcf35544-c072-4fe6-852c-7dee622564b2" (UID: "bcf35544-c072-4fe6-852c-7dee622564b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:00.775132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.775103    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca" pod "bcf35544-c072-4fe6-852c-7dee622564b2" (UID: "bcf35544-c072-4fe6-852c-7dee622564b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:50:00.775258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.775212    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5" pod "bcf35544-c072-4fe6-852c-7dee622564b2" (UID: "bcf35544-c072-4fe6-852c-7dee622564b2"). InnerVolumeSpecName "kube-api-access-j5ng5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:00.874091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.874053    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:00.874091 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.874088    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf35544-c072-4fe6-852c-7dee622564b2-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:00.874324 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:00.874104    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ng5\" (UniqueName: \"kubernetes.io/projected/bcf35544-c072-4fe6-852c-7dee622564b2-kube-api-access-j5ng5\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:01.491516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:01.491477    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"
Nov 02 20:50:01.491516 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:01.491505    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89" event={"ID":"bcf35544-c072-4fe6-852c-7dee622564b2","Type":"ContainerDied","Data":"be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e"}
Nov 02 20:50:01.491733 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:01.491555    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2efe3f9c0f1eaf49e5453687ce472517dba4b0623a3ad0b4317f7f029d5b2e"
Nov 02 20:50:16.094290 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:50:16.094252    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:50:29.782230 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.782187    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"]
Nov 02 20:50:29.830754 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.830715    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sm89"]
Nov 02 20:50:29.897995 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.897966    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"]
Nov 02 20:50:29.898436 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.898420    5099 cpu_manager.go:414] "containerMap: removing container" podUID="bcf35544-c072-4fe6-852c-7dee622564b2" containerName="revoke-keys"
Nov 02 20:50:29.898496 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.898437    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf35544-c072-4fe6-852c-7dee622564b2" containerName="revoke-keys"
Nov 02 20:50:29.898560 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.898550    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf35544-c072-4fe6-852c-7dee622564b2" containerName="revoke-keys"
Nov 02 20:50:29.901858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.901837    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:29.952155 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.952118    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:50:29.952361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.952269    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:50:29.955678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.955643    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"]
Nov 02 20:50:29.973087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.973051    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:29.973287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.973101    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qpf\" (UniqueName: \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:29.973287 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:29.973224    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.074762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.074651    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.074762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.074696    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qpf\" (UniqueName: \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.074762 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.074719    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.075389 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.075365    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.077359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.077333    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.128487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.128450    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qpf\" (UniqueName: \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\") pod \"maas-api-revoke-sub-3d4156f1-9hgkv\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.215696 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.215656    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:30.581857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.581825    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"]
Nov 02 20:50:30.583050 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:50:30.583023    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05ef380_b672_45aa_8d82_ce6affe745c6.slice/crio-9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f WatchSource:0}: Error finding container 9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f: Status 404 returned error can't find the container with id 9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f
Nov 02 20:50:30.739774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:30.739731    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv" event={"ID":"e05ef380-b672-45aa-8d82-ce6affe745c6","Type":"ContainerStarted","Data":"9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f"}
Nov 02 20:50:31.256108 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:31.256073    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf35544-c072-4fe6-852c-7dee622564b2" path="/var/lib/kubelet/pods/bcf35544-c072-4fe6-852c-7dee622564b2/volumes"
Nov 02 20:50:31.746923 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:31.746892    5099 generic.go:356] "Generic (PLEG): container finished" podID="e05ef380-b672-45aa-8d82-ce6affe745c6" containerID="46b438765214f4b1b433b727a8184638f139b10d47a5f1cd34be48ea688114f5" exitCode=0
Nov 02 20:50:31.747110 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:31.746937    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv" event={"ID":"e05ef380-b672-45aa-8d82-ce6affe745c6","Type":"ContainerDied","Data":"46b438765214f4b1b433b727a8184638f139b10d47a5f1cd34be48ea688114f5"}
Nov 02 20:50:32.892953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:32.892928    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:33.005964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.005713    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\") pod \"e05ef380-b672-45aa-8d82-ce6affe745c6\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") "
Nov 02 20:50:33.005964 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.005850    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\") pod \"e05ef380-b672-45aa-8d82-ce6affe745c6\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") "
Nov 02 20:50:33.006203 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.006017    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\" (UniqueName: \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\") pod \"e05ef380-b672-45aa-8d82-ce6affe745c6\" (UID: \"e05ef380-b672-45aa-8d82-ce6affe745c6\") "
Nov 02 20:50:33.006499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.006478    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca" pod "e05ef380-b672-45aa-8d82-ce6affe745c6" (UID: "e05ef380-b672-45aa-8d82-ce6affe745c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:50:33.008859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.008832    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca" pod "e05ef380-b672-45aa-8d82-ce6affe745c6" (UID: "e05ef380-b672-45aa-8d82-ce6affe745c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:50:33.008981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.008950    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf" pod "e05ef380-b672-45aa-8d82-ce6affe745c6" (UID: "e05ef380-b672-45aa-8d82-ce6affe745c6"). InnerVolumeSpecName "kube-api-access-s2qpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:50:33.107674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.107641    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:33.107674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.107671    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qpf\" (UniqueName: \"kubernetes.io/projected/e05ef380-b672-45aa-8d82-ce6affe745c6-kube-api-access-s2qpf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:33.107674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.107681    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05ef380-b672-45aa-8d82-ce6affe745c6-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:50:33.758870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.758817    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv" event={"ID":"e05ef380-b672-45aa-8d82-ce6affe745c6","Type":"ContainerDied","Data":"9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f"}
Nov 02 20:50:33.758870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.758885    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae1340d1df9fdee58d27f73159c7f55c075a38194423edb2bbcfa4a70dc648f"
Nov 02 20:50:33.759096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:33.758831    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"
Nov 02 20:50:46.092953 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:50:46.092912    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:50:56.493396 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:56.493362    5099 scope.go:122] "RemoveContainer" containerID="a14d8eae165b2cc0ee4349150996df8f71f984d8e816f7162577fcac46ee7baa"
Nov 02 20:50:56.502328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:56.502303    5099 scope.go:122] "RemoveContainer" containerID="5dde0d73546b9723880a1fa1a62b9ed214da31bbf906b50e0baf8521dd27e184"
Nov 02 20:50:59.043714 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:59.043673    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"]
Nov 02 20:50:59.089467 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:59.089432    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-d8ph9"]
Nov 02 20:50:59.256081 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:50:59.256050    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9be8f92-02fc-42bc-b285-f25f03055de4" path="/var/lib/kubelet/pods/c9be8f92-02fc-42bc-b285-f25f03055de4/volumes"
Nov 02 20:51:16.093709 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:51:16.093666    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:51:33.043571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:33.043533    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"]
Nov 02 20:51:33.090643 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:33.090609    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rnd5g"]
Nov 02 20:51:33.256653 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:33.256621    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53d1deb-c928-4737-9d34-71daf35d7de3" path="/var/lib/kubelet/pods/e53d1deb-c928-4737-9d34-71daf35d7de3/volumes"
Nov 02 20:51:39.798098 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.798059    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"]
Nov 02 20:51:39.798525 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.798512    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e05ef380-b672-45aa-8d82-ce6affe745c6" containerName="revoke-keys"
Nov 02 20:51:39.798571 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.798526    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05ef380-b672-45aa-8d82-ce6affe745c6" containerName="revoke-keys"
Nov 02 20:51:39.798625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.798616    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05ef380-b672-45aa-8d82-ce6affe745c6" containerName="revoke-keys"
Nov 02 20:51:39.802150 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.802134    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.847304 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.847274    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.847479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.847310    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.847479 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.847414    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ksz\" (UniqueName: \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.854402 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.854371    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"]
Nov 02 20:51:39.898505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.898473    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:51:39.898682 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.898663    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:51:39.948647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.948615    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.948857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.948653    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.948857 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.948710    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ksz\" (UniqueName: \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.949309 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.949273    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:39.951057 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:39.951037    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:40.002914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:40.002879    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ksz\" (UniqueName: \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\") pod \"maas-api-revoke-sub-e2fc9c5b-bt7qw\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:40.116125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:40.116082    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:40.481390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:40.481359    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"]
Nov 02 20:51:40.482286 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:51:40.482253    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201be22_a76b_46dd_823c_ed174e80b366.slice/crio-9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17 WatchSource:0}: Error finding container 9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17: Status 404 returned error can't find the container with id 9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17
Nov 02 20:51:41.159640 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:41.159607    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw" event={"ID":"9201be22-a76b-46dd-823c-ed174e80b366","Type":"ContainerStarted","Data":"2133e28b837fdbd03ae731e7734fb98ffaa06a8c8f5348ef2110c3cd8070ece2"}
Nov 02 20:51:41.160088 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:41.159661    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw" event={"ID":"9201be22-a76b-46dd-823c-ed174e80b366","Type":"ContainerStarted","Data":"9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17"}
Nov 02 20:51:41.267870 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:41.267820    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw" podStartSLOduration=2.267806897 podStartE2EDuration="2.267806897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:51:41.266338647 +0000 UTC m=+3590.140609537" watchObservedRunningTime="2026-11-02 20:51:41.267806897 +0000 UTC m=+3590.142077782"
Nov 02 20:51:42.166658 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:42.166621    5099 generic.go:356] "Generic (PLEG): container finished" podID="9201be22-a76b-46dd-823c-ed174e80b366" containerID="2133e28b837fdbd03ae731e7734fb98ffaa06a8c8f5348ef2110c3cd8070ece2" exitCode=0
Nov 02 20:51:42.167087 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:42.166700    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw" event={"ID":"9201be22-a76b-46dd-823c-ed174e80b366","Type":"ContainerDied","Data":"2133e28b837fdbd03ae731e7734fb98ffaa06a8c8f5348ef2110c3cd8070ece2"}
Nov 02 20:51:43.318361 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.318334    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:43.383470 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.383436    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\" (UniqueName: \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\") pod \"9201be22-a76b-46dd-823c-ed174e80b366\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") "
Nov 02 20:51:43.383665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.383476    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\") pod \"9201be22-a76b-46dd-823c-ed174e80b366\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") "
Nov 02 20:51:43.383665 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.383631    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\") pod \"9201be22-a76b-46dd-823c-ed174e80b366\" (UID: \"9201be22-a76b-46dd-823c-ed174e80b366\") "
Nov 02 20:51:43.383978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.383952    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca" pod "9201be22-a76b-46dd-823c-ed174e80b366" (UID: "9201be22-a76b-46dd-823c-ed174e80b366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:51:43.385659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.385634    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca" pod "9201be22-a76b-46dd-823c-ed174e80b366" (UID: "9201be22-a76b-46dd-823c-ed174e80b366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:51:43.385793 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.385703    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz" pod "9201be22-a76b-46dd-823c-ed174e80b366" (UID: "9201be22-a76b-46dd-823c-ed174e80b366"). InnerVolumeSpecName "kube-api-access-z6ksz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:51:43.485359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.485271    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:43.485359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.485307    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ksz\" (UniqueName: \"kubernetes.io/projected/9201be22-a76b-46dd-823c-ed174e80b366-kube-api-access-z6ksz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:43.485359 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:43.485318    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9201be22-a76b-46dd-823c-ed174e80b366-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:44.179897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:44.179864    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw" event={"ID":"9201be22-a76b-46dd-823c-ed174e80b366","Type":"ContainerDied","Data":"9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17"}
Nov 02 20:51:44.179897 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:44.179904    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"
Nov 02 20:51:44.180109 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:44.179912    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e3059d186f9e67ea8011a51a41d35dca9686d050ec0374f6cd6755c31025f17"
Nov 02 20:51:45.042655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:45.042620    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"]
Nov 02 20:51:45.090096 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:45.090063    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pj5js"]
Nov 02 20:51:45.256036 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:45.256006    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edf0467-6f8a-471f-8158-9b9434f6902a" path="/var/lib/kubelet/pods/1edf0467-6f8a-471f-8158-9b9434f6902a/volumes"
Nov 02 20:51:46.093882 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:51:46.093839    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:51:50.519291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.519246    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"]
Nov 02 20:51:50.519954 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.519933    5099 cpu_manager.go:414] "containerMap: removing container" podUID="9201be22-a76b-46dd-823c-ed174e80b366" containerName="revoke-keys"
Nov 02 20:51:50.520066 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.519955    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9201be22-a76b-46dd-823c-ed174e80b366" containerName="revoke-keys"
Nov 02 20:51:50.520132 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.520091    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="9201be22-a76b-46dd-823c-ed174e80b366" containerName="revoke-keys"
Nov 02 20:51:50.530994 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.530965    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.575577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.575547    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"]
Nov 02 20:51:50.619231 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.619204    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:51:50.619390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.619265    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:51:50.652839 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.652804    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.653012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.652907    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqfq\" (UniqueName: \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.653012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.652937    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.753501 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.753459    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.753697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.753528    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqfq\" (UniqueName: \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.753697 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.753548    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.754267 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.754247    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.755989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.755967    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.809024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.808954    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqfq\" (UniqueName: \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\") pod \"maas-api-revoke-sub-5a29efb3-8t9sj\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:50.843981 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:50.843946    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:51.207154 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:51.207121    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"]
Nov 02 20:51:51.221055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:51.221028    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj" event={"ID":"5ea9366b-0dda-475f-82ce-6f23c15d025f","Type":"ContainerStarted","Data":"6a71227568d6321d10baefca831a60fbdae8e4bb3ac1f72be532d7aef3d96fb5"}
Nov 02 20:51:52.052577 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:52.052551    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:51:52.099140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:52.099110    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:51:52.228492 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:52.228460    5099 generic.go:356] "Generic (PLEG): container finished" podID="5ea9366b-0dda-475f-82ce-6f23c15d025f" containerID="123a57ba60fbe692a1b5524227a71cfbc55e2c2cec7306a5b1252968b7284c98" exitCode=0
Nov 02 20:51:52.228659 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:52.228498    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj" event={"ID":"5ea9366b-0dda-475f-82ce-6f23c15d025f","Type":"ContainerDied","Data":"123a57ba60fbe692a1b5524227a71cfbc55e2c2cec7306a5b1252968b7284c98"}
Nov 02 20:51:53.437248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.437217    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:53.597609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.597491    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\") pod \"5ea9366b-0dda-475f-82ce-6f23c15d025f\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") "
Nov 02 20:51:53.597609 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.597557    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\" (UniqueName: \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\") pod \"5ea9366b-0dda-475f-82ce-6f23c15d025f\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") "
Nov 02 20:51:53.597908 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.597641    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\") pod \"5ea9366b-0dda-475f-82ce-6f23c15d025f\" (UID: \"5ea9366b-0dda-475f-82ce-6f23c15d025f\") "
Nov 02 20:51:53.598421 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.598384    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca" pod "5ea9366b-0dda-475f-82ce-6f23c15d025f" (UID: "5ea9366b-0dda-475f-82ce-6f23c15d025f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:51:53.600913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.600776    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca" pod "5ea9366b-0dda-475f-82ce-6f23c15d025f" (UID: "5ea9366b-0dda-475f-82ce-6f23c15d025f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:51:53.600913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.600875    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq" pod "5ea9366b-0dda-475f-82ce-6f23c15d025f" (UID: "5ea9366b-0dda-475f-82ce-6f23c15d025f"). InnerVolumeSpecName "kube-api-access-mcqfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:51:53.699246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.699200    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:53.699246 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.699243    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqfq\" (UniqueName: \"kubernetes.io/projected/5ea9366b-0dda-475f-82ce-6f23c15d025f-kube-api-access-mcqfq\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:53.699536 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:53.699261    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea9366b-0dda-475f-82ce-6f23c15d025f-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:51:54.278178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:54.278141    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj" event={"ID":"5ea9366b-0dda-475f-82ce-6f23c15d025f","Type":"ContainerDied","Data":"6a71227568d6321d10baefca831a60fbdae8e4bb3ac1f72be532d7aef3d96fb5"}
Nov 02 20:51:54.278460 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:54.278439    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a71227568d6321d10baefca831a60fbdae8e4bb3ac1f72be532d7aef3d96fb5"
Nov 02 20:51:54.278644 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:54.278628    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"
Nov 02 20:51:55.263808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:55.254588    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:51:55.263808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:55.254646    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:51:55.418808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:55.416942    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:51:55.418808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:55.417021    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:51:56.527116 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:51:56.527083    5099 scope.go:122] "RemoveContainer" containerID="93f2fc46970b06e0d3b7c5f7fa8ea8d3a533e1c43da2320439926848a32dfb1b"
Nov 02 20:52:16.093656 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:52:16.093616    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:52:17.067647 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.067612    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"]
Nov 02 20:52:17.114798 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.114748    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9hgkv"]
Nov 02 20:52:17.181339 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.181305    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"]
Nov 02 20:52:17.181883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.181862    5099 cpu_manager.go:414] "containerMap: removing container" podUID="5ea9366b-0dda-475f-82ce-6f23c15d025f" containerName="revoke-keys"
Nov 02 20:52:17.181883 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.181882    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea9366b-0dda-475f-82ce-6f23c15d025f" containerName="revoke-keys"
Nov 02 20:52:17.182075 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.182057    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea9366b-0dda-475f-82ce-6f23c15d025f" containerName="revoke-keys"
Nov 02 20:52:17.186920 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.186898    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.237901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.237855    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:52:17.238276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.238249    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"]
Nov 02 20:52:17.238486 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.238466    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:52:17.260181 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.260145    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05ef380-b672-45aa-8d82-ce6affe745c6" path="/var/lib/kubelet/pods/e05ef380-b672-45aa-8d82-ce6affe745c6/volumes"
Nov 02 20:52:17.289004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.288968    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgwh\" (UniqueName: \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.289192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.289152    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.289271 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.289253    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.390115 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.390076    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.390325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.390134    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.390325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.390245    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgwh\" (UniqueName: \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.390864 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.390845    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.392539 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.392519    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.443856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.443825    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgwh\" (UniqueName: \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\") pod \"maas-api-revoke-sub-3d4156f1-724lx\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.500294 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.500264    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:17.863065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:17.863034    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"]
Nov 02 20:52:17.863776 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:52:17.863746    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237d2f6e_9290_4ed8_bbdd_c9f15f958f6c.slice/crio-6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134 WatchSource:0}: Error finding container 6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134: Status 404 returned error can't find the container with id 6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134
Nov 02 20:52:18.450377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:18.450285    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx" event={"ID":"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c","Type":"ContainerStarted","Data":"fa4e8b93f48db5ab8fbc5db920d4ed14cb358bf46ccc5664d7fcd90bc83e61e5"}
Nov 02 20:52:18.450377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:18.450348    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx" event={"ID":"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c","Type":"ContainerStarted","Data":"6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134"}
Nov 02 20:52:18.557978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:18.557921    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx" podStartSLOduration=1.557905725 podStartE2EDuration="1.557905725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:18.556818319 +0000 UTC m=+3627.431089208" watchObservedRunningTime="2026-11-02 20:52:18.557905725 +0000 UTC m=+3627.432176609"
Nov 02 20:52:19.457431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:19.457390    5099 generic.go:356] "Generic (PLEG): container finished" podID="237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" containerID="fa4e8b93f48db5ab8fbc5db920d4ed14cb358bf46ccc5664d7fcd90bc83e61e5" exitCode=0
Nov 02 20:52:19.457851 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:19.457475    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx" event={"ID":"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c","Type":"ContainerDied","Data":"fa4e8b93f48db5ab8fbc5db920d4ed14cb358bf46ccc5664d7fcd90bc83e61e5"}
Nov 02 20:52:20.606996 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.606971    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:20.724593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.724504    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\" (UniqueName: \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\") pod \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") "
Nov 02 20:52:20.724593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.724566    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\") pod \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") "
Nov 02 20:52:20.724893 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.724657    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\") pod \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\" (UID: \"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c\") "
Nov 02 20:52:20.725082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.725050    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca" pod "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" (UID: "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:52:20.727180 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.727153    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca" pod "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" (UID: "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:52:20.727264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.727161    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh" pod "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" (UID: "237d2f6e-9290-4ed8-bbdd-c9f15f958f6c"). InnerVolumeSpecName "kube-api-access-gqgwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:52:20.826092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.826059    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:52:20.826092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.826088    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgwh\" (UniqueName: \"kubernetes.io/projected/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-kube-api-access-gqgwh\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:52:20.826092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:20.826098    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:52:21.470502 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:21.470473    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx" event={"ID":"237d2f6e-9290-4ed8-bbdd-c9f15f958f6c","Type":"ContainerDied","Data":"6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134"}
Nov 02 20:52:21.470674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:21.470510    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"
Nov 02 20:52:21.470674 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:21.470517    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4356e668f1b11a44830d51c389af59fd96713e524f2b069cc7e35ffe2bf134"
Nov 02 20:52:30.042586 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:30.042506    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"]
Nov 02 20:52:30.089956 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:30.089923    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dpxpk"]
Nov 02 20:52:31.256504 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:31.256467    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b69a301-6e8b-43d2-b702-7722a769ad50" path="/var/lib/kubelet/pods/0b69a301-6e8b-43d2-b702-7722a769ad50/volumes"
Nov 02 20:52:42.044192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:42.044155    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"]
Nov 02 20:52:42.090808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:42.090753    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qg64t"]
Nov 02 20:52:43.256915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:43.256879    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6a7fcd-8645-41ab-a435-5a907b84e0c8" path="/var/lib/kubelet/pods/ee6a7fcd-8645-41ab-a435-5a907b84e0c8/volumes"
Nov 02 20:52:46.096118 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:52:46.096079    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:52:56.549982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:56.549942    5099 scope.go:122] "RemoveContainer" containerID="418a244855194ec81055ae145d7db6fe0a00a2d72d6a135a153e210bb47bb744"
Nov 02 20:52:56.559308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:56.559288    5099 scope.go:122] "RemoveContainer" containerID="2bfef1c985ef0e674b2c274bbcb020b4968747dde1350919a8fad23139aea19f"
Nov 02 20:52:57.562764 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.562731    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:52:57.563215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.563202    5099 cpu_manager.go:414] "containerMap: removing container" podUID="237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" containerName="revoke-keys"
Nov 02 20:52:57.563269 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.563216    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" containerName="revoke-keys"
Nov 02 20:52:57.563306 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.563290    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" containerName="revoke-keys"
Nov 02 20:52:57.567901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.567880    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.618952 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.618916    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:52:57.655769 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.655725    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8h2\" (UniqueName: \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.656261 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.656229    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.665042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.664842    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0a02f7\""
Nov 02 20:52:57.665042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.664999    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-918279b3-dockercfg-m7zfz\""
Nov 02 20:52:57.714681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.714649    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:52:57.718456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.718430    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.757071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.757039    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.757235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.757090    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.757235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.757114    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8h2\" (UniqueName: \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.757235 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.757150    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xd46\" (UniqueName: \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.757730 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.757709    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.765997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.765969    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:52:57.810993 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.810968    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8h2\" (UniqueName: \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\") pod \"payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.857759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.857687    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.857759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.857752    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd46\" (UniqueName: \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.858431 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.858406    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:57.881617 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.881585    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:57.911573 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:57.911537    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd46\" (UniqueName: \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\") pod \"payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:58.032004 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.031970    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:58.249535 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.247987    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:52:58.249535 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:52:58.249410    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afe197e_a5d9_406c_b52d_5c2b707a6857.slice/crio-5d777156a2285524ff899b118b0a03426200ec32b2a64e0f9dc7921a111af615 WatchSource:0}: Error finding container 5d777156a2285524ff899b118b0a03426200ec32b2a64e0f9dc7921a111af615: Status 404 returned error can't find the container with id 5d777156a2285524ff899b118b0a03426200ec32b2a64e0f9dc7921a111af615
Nov 02 20:52:58.395623 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.395600    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:52:58.396656 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:52:58.396628    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b4a54e_344e_4bf2_ab08_48b0724a4242.slice/crio-31689d721e41a89ef873f485d3043c4bce2aaf0317d397fc9c90cfcca807acb3 WatchSource:0}: Error finding container 31689d721e41a89ef873f485d3043c4bce2aaf0317d397fc9c90cfcca807acb3: Status 404 returned error can't find the container with id 31689d721e41a89ef873f485d3043c4bce2aaf0317d397fc9c90cfcca807acb3
Nov 02 20:52:58.708405 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.708367    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" event={"ID":"0afe197e-a5d9-406c-b52d-5c2b707a6857","Type":"ContainerStarted","Data":"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"}
Nov 02 20:52:58.708910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.708435    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:52:58.708910 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.708461    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" event={"ID":"0afe197e-a5d9-406c-b52d-5c2b707a6857","Type":"ContainerStarted","Data":"5d777156a2285524ff899b118b0a03426200ec32b2a64e0f9dc7921a111af615"}
Nov 02 20:52:58.709965 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.709942    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" event={"ID":"43b4a54e-344e-4bf2-ab08-48b0724a4242","Type":"ContainerStarted","Data":"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"}
Nov 02 20:52:58.710077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.709980    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" event={"ID":"43b4a54e-344e-4bf2-ab08-48b0724a4242","Type":"ContainerStarted","Data":"31689d721e41a89ef873f485d3043c4bce2aaf0317d397fc9c90cfcca807acb3"}
Nov 02 20:52:58.710077 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.710008    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:52:58.815128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.815069    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" podStartSLOduration=1.815053694 podStartE2EDuration="1.815053694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:58.814633547 +0000 UTC m=+3667.688904436" watchObservedRunningTime="2026-11-02 20:52:58.815053694 +0000 UTC m=+3667.689324615"
Nov 02 20:52:58.921464 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:58.921398    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" podStartSLOduration=1.9213794499999999 podStartE2EDuration="1.92137945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:58.921137472 +0000 UTC m=+3667.795408361" watchObservedRunningTime="2026-11-02 20:52:58.92137945 +0000 UTC m=+3667.795650341"
Nov 02 20:52:59.701123 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.701084    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:52:59.705012 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.704991    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.757951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.757911    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:52:59.777249 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.777206    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.777445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.777280    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.777498 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.777457    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfff\" (UniqueName: \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.777625 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.777606    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.801953 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.801917    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0a02f7-serving-cert\""
Nov 02 20:52:59.802141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.801999    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\""
Nov 02 20:52:59.802141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.802021    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 20:52:59.878565 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.878530    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.878775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.878584    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.878775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.878618    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfff\" (UniqueName: \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.878775 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.878663    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.878775 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:52:59.878681    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-serving-cert: secret "maas-api-e2e-worker-single-main-0a02f7-serving-cert" not found
Nov 02 20:52:59.878775 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:52:59.878744    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls podName:b9b3a1df-e741-495a-9cfa-d4e511fbd841 nodeName:}" failed. No retries permitted until 2026-11-02 20:53:00.378728205 +0000 UTC m=+3669.252999071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls") pod "maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" (UID: "b9b3a1df-e741-495a-9cfa-d4e511fbd841") : secret "maas-api-e2e-worker-single-main-0a02f7-serving-cert" not found
Nov 02 20:52:59.879308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.879288    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.881186 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.881164    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:52:59.932982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:52:59.932948    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfff\" (UniqueName: \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:00.383973 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:00.383924    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:00.386400 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:00.386369    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:00.620117 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:00.620078    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:00.983614 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:00.983583    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:53:00.985033 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:53:00.985003    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b3a1df_e741_495a_9cfa_d4e511fbd841.slice/crio-db2728b53da81466ded028e0bdbc31d8316b55024654a2d63656aaba15d4d324 WatchSource:0}: Error finding container db2728b53da81466ded028e0bdbc31d8316b55024654a2d63656aaba15d4d324: Status 404 returned error can't find the container with id db2728b53da81466ded028e0bdbc31d8316b55024654a2d63656aaba15d4d324
Nov 02 20:53:00.986590 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:00.986570    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 20:53:01.730071 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:01.730034    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" event={"ID":"b9b3a1df-e741-495a-9cfa-d4e511fbd841","Type":"ContainerStarted","Data":"db2728b53da81466ded028e0bdbc31d8316b55024654a2d63656aaba15d4d324"}
Nov 02 20:53:02.740455 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:02.740411    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" event={"ID":"b9b3a1df-e741-495a-9cfa-d4e511fbd841","Type":"ContainerStarted","Data":"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"}
Nov 02 20:53:02.740963 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:02.740566    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:02.849858 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:02.849804    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" podStartSLOduration=2.9861012479999998 podStartE2EDuration="3.849777218s" totalImagesPullingTime="863.67597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:53:02.849308971 +0000 UTC m=+3671.723579858" watchObservedRunningTime="2026-11-02 20:53:02.849777218 +0000 UTC m=+3671.724048105"
Nov 02 20:53:08.752345 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:08.752315    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:53:09.722844 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:09.722812    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:53:09.723046 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:09.722882    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:53:15.045011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:15.044976    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"]
Nov 02 20:53:15.090241 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:15.090208    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2r4l2"]
Nov 02 20:53:15.257548 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:15.257508    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dd6492-3b74-448a-91aa-c13ca5cd8601" path="/var/lib/kubelet/pods/c7dd6492-3b74-448a-91aa-c13ca5cd8601/volumes"
Nov 02 20:53:16.094183 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:53:16.094139    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:53:43.043557 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:43.043523    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"]
Nov 02 20:53:43.090530 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:43.090494    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpbj8"]
Nov 02 20:53:43.256328 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:43.256290    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05b67f3-76f6-4c52-810f-b566a579399b" path="/var/lib/kubelet/pods/e05b67f3-76f6-4c52-810f-b566a579399b/volumes"
Nov 02 20:53:46.094039 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:53:46.093982    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:53:56.584572 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:56.584539    5099 scope.go:122] "RemoveContainer" containerID="c850468b4618400f595e84701a725e40f5058df776f4be23acb31f22fcd9dd43"
Nov 02 20:53:56.595881 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:53:56.594961    5099 scope.go:122] "RemoveContainer" containerID="38e2bea41b58754a7c0cb3b1b14684ef324475f4dd4a9cde63ed284f4f186c8e"
Nov 02 20:54:16.093540 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:54:16.093453    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:54:17.043390 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:17.043355    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"]
Nov 02 20:54:17.090613 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:17.090578    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2srk2"]
Nov 02 20:54:17.257179 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:17.257146    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f41515-4fd4-4c62-a50f-72fb73ad62b1" path="/var/lib/kubelet/pods/42f41515-4fd4-4c62-a50f-72fb73ad62b1/volumes"
Nov 02 20:54:40.111639 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.111605    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"]
Nov 02 20:54:40.158951 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.158920    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hl68c"]
Nov 02 20:54:40.334830 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.334777    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:54:40.335068 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.335041    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" podUID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" containerName="maas-api" containerID="cri-o://386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7" gracePeriod=30
Nov 02 20:54:40.573774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.573677    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:54:40.692037 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:40.691578    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:54:41.272056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.272015    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0442ea78-76ac-4032-a463-68194a68e950" path="/var/lib/kubelet/pods/0442ea78-76ac-4032-a463-68194a68e950/volumes"
Nov 02 20:54:41.597949 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.597926    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:54:41.673880 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.673842    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\") pod \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") "
Nov 02 20:54:41.674079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.673926    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") pod \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") "
Nov 02 20:54:41.674079 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.673996    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\") pod \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") "
Nov 02 20:54:41.674223 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.674081    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\" (UniqueName: \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\") pod \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\" (UID: \"b9b3a1df-e741-495a-9cfa-d4e511fbd841\") "
Nov 02 20:54:41.674308 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.674278    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca" pod "b9b3a1df-e741-495a-9cfa-d4e511fbd841" (UID: "b9b3a1df-e741-495a-9cfa-d4e511fbd841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:54:41.674440 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.674423    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b3a1df-e741-495a-9cfa-d4e511fbd841-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:41.676340 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.676316    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls" pod "b9b3a1df-e741-495a-9cfa-d4e511fbd841" (UID: "b9b3a1df-e741-495a-9cfa-d4e511fbd841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:54:41.676652 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.676631    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff" pod "b9b3a1df-e741-495a-9cfa-d4e511fbd841" (UID: "b9b3a1df-e741-495a-9cfa-d4e511fbd841"). InnerVolumeSpecName "kube-api-access-xtfff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:54:41.676856 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.676839    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs" pod "b9b3a1df-e741-495a-9cfa-d4e511fbd841" (UID: "b9b3a1df-e741-495a-9cfa-d4e511fbd841"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:54:41.775804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.775744    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:41.775804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.775776    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9b3a1df-e741-495a-9cfa-d4e511fbd841-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:41.776024 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:41.775822    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfff\" (UniqueName: \"kubernetes.io/projected/b9b3a1df-e741-495a-9cfa-d4e511fbd841-kube-api-access-xtfff\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:54:42.318526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.318495    5099 generic.go:356] "Generic (PLEG): container finished" podID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" containerID="386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7" exitCode=0
Nov 02 20:54:42.319011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.318533    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"
Nov 02 20:54:42.319011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.318548    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" event={"ID":"b9b3a1df-e741-495a-9cfa-d4e511fbd841","Type":"ContainerDied","Data":"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"}
Nov 02 20:54:42.319011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.318591    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9" event={"ID":"b9b3a1df-e741-495a-9cfa-d4e511fbd841","Type":"ContainerDied","Data":"db2728b53da81466ded028e0bdbc31d8316b55024654a2d63656aaba15d4d324"}
Nov 02 20:54:42.319011 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.318614    5099 scope.go:122] "RemoveContainer" containerID="386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"
Nov 02 20:54:42.330023 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.330003    5099 scope.go:122] "RemoveContainer" containerID="386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"
Nov 02 20:54:42.330317 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:54:42.330286    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7\": container with ID starting with 386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7 not found: ID does not exist" containerID="386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"
Nov 02 20:54:42.330366 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.330329    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7"} err="failed to get container status \"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7\": rpc error: code = NotFound desc = could not find container \"386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7\": container with ID starting with 386a212ede69a4555d327a07b9b2c42bf434b2dd0d54484c8fd940991ffba8a7 not found: ID does not exist"
Nov 02 20:54:42.473286 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.473248    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:54:42.520228 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:42.520198    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0a02f7-747d58776-vbcc9"]
Nov 02 20:54:43.256519 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:43.256482    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" path="/var/lib/kubelet/pods/b9b3a1df-e741-495a-9cfa-d4e511fbd841/volumes"
Nov 02 20:54:45.574662 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:45.574616    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing" containerID="cri-o://24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510" gracePeriod=25
Nov 02 20:54:45.692567 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:45.692520    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing" containerID="cri-o://41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf" gracePeriod=25
Nov 02 20:54:46.094089 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:54:46.094053    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:54:49.721025 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:49.720987    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:54:49.721452 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:49.720986    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:54:53.496607 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:53.496575    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"]
Nov 02 20:54:53.543594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:53.543554    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-724lx"]
Nov 02 20:54:55.256593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:55.256561    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237d2f6e-9290-4ed8-bbdd-c9f15f958f6c" path="/var/lib/kubelet/pods/237d2f6e-9290-4ed8-bbdd-c9f15f958f6c/volumes"
Nov 02 20:54:56.623140 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:56.623111    5099 scope.go:122] "RemoveContainer" containerID="fd3762abb527b6f15fad76d7a92c9a7c1be40fbf2c2bd91983e2236375862738"
Nov 02 20:54:56.631604 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:56.631586    5099 scope.go:122] "RemoveContainer" containerID="aa1329192f9035c4cb5d670d55f11fdb84571c174b860dcb794d03df0f59dd15"
Nov 02 20:54:59.720507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:59.720464    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:54:59.720507 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:54:59.720493    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:55:05.848691 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.848631    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:55:05.920901 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.920856    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\" (UniqueName: \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\") pod \"43b4a54e-344e-4bf2-ab08-48b0724a4242\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") "
Nov 02 20:55:05.921114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.921007    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\") pod \"43b4a54e-344e-4bf2-ab08-48b0724a4242\" (UID: \"43b4a54e-344e-4bf2-ab08-48b0724a4242\") "
Nov 02 20:55:05.921425 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.921398    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume" pod "43b4a54e-344e-4bf2-ab08-48b0724a4242" (UID: "43b4a54e-344e-4bf2-ab08-48b0724a4242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:55:05.923043 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.923015    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46" pod "43b4a54e-344e-4bf2-ab08-48b0724a4242" (UID: "43b4a54e-344e-4bf2-ab08-48b0724a4242"). InnerVolumeSpecName "kube-api-access-5xd46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:55:05.940264 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:05.940244    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:55:06.022499 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.022466    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\" (UniqueName: \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\") pod \"0afe197e-a5d9-406c-b52d-5c2b707a6857\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") "
Nov 02 20:55:06.022670 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.022570    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\") pod \"0afe197e-a5d9-406c-b52d-5c2b707a6857\" (UID: \"0afe197e-a5d9-406c-b52d-5c2b707a6857\") "
Nov 02 20:55:06.022804 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.022768    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xd46\" (UniqueName: \"kubernetes.io/projected/43b4a54e-344e-4bf2-ab08-48b0724a4242-kube-api-access-5xd46\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:06.022852 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.022808    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b4a54e-344e-4bf2-ab08-48b0724a4242-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:06.022919 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.022896    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume" pod "0afe197e-a5d9-406c-b52d-5c2b707a6857" (UID: "0afe197e-a5d9-406c-b52d-5c2b707a6857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:55:06.024551 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.024532    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2" pod "0afe197e-a5d9-406c-b52d-5c2b707a6857" (UID: "0afe197e-a5d9-406c-b52d-5c2b707a6857"). InnerVolumeSpecName "kube-api-access-rv8h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:55:06.123482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.123444    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8h2\" (UniqueName: \"kubernetes.io/projected/0afe197e-a5d9-406c-b52d-5c2b707a6857-kube-api-access-rv8h2\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:06.123482 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.123477    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0afe197e-a5d9-406c-b52d-5c2b707a6857-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:06.446540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.446456    5099 generic.go:356] "Generic (PLEG): container finished" podID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerID="41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf" exitCode=0
Nov 02 20:55:06.446540 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.446502    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"
Nov 02 20:55:06.446771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.446536    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" event={"ID":"0afe197e-a5d9-406c-b52d-5c2b707a6857","Type":"ContainerDied","Data":"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"}
Nov 02 20:55:06.446771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.446591    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9" event={"ID":"0afe197e-a5d9-406c-b52d-5c2b707a6857","Type":"ContainerDied","Data":"5d777156a2285524ff899b118b0a03426200ec32b2a64e0f9dc7921a111af615"}
Nov 02 20:55:06.446771 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.446622    5099 scope.go:122] "RemoveContainer" containerID="41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"
Nov 02 20:55:06.448056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.448031    5099 generic.go:356] "Generic (PLEG): container finished" podID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerID="24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510" exitCode=0
Nov 02 20:55:06.448141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.448083    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"
Nov 02 20:55:06.448141 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.448112    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" event={"ID":"43b4a54e-344e-4bf2-ab08-48b0724a4242","Type":"ContainerDied","Data":"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"}
Nov 02 20:55:06.448224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.448149    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k" event={"ID":"43b4a54e-344e-4bf2-ab08-48b0724a4242","Type":"ContainerDied","Data":"31689d721e41a89ef873f485d3043c4bce2aaf0317d397fc9c90cfcca807acb3"}
Nov 02 20:55:06.457055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.457037    5099 scope.go:122] "RemoveContainer" containerID="41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"
Nov 02 20:55:06.457375 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:55:06.457355    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf\": container with ID starting with 41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf not found: ID does not exist" containerID="41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"
Nov 02 20:55:06.457450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.457382    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf"} err="failed to get container status \"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf\": rpc error: code = NotFound desc = could not find container \"41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf\": container with ID starting with 41d37c955d878f98678d4ec295ccad8bbcd911fd4412cd874303c2866906ccdf not found: ID does not exist"
Nov 02 20:55:06.457450 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.457398    5099 scope.go:122] "RemoveContainer" containerID="24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"
Nov 02 20:55:06.465505 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.465487    5099 scope.go:122] "RemoveContainer" containerID="24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"
Nov 02 20:55:06.465771 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:55:06.465747    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510\": container with ID starting with 24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510 not found: ID does not exist" containerID="24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"
Nov 02 20:55:06.465873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.465795    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510"} err="failed to get container status \"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510\": rpc error: code = NotFound desc = could not find container \"24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510\": container with ID starting with 24c6f0f8c972682e6e9059e8a559f34743fb2e82a3b2ba35b6edc15fe006a510 not found: ID does not exist"
Nov 02 20:55:06.601426 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.601398    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:55:06.649594 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.649564    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0a02f7-559dft7zx9"]
Nov 02 20:55:06.707377 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.707295    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:55:06.710712 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:06.710687    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0a02f7-74959c5979gd8k"]
Nov 02 20:55:07.257073 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:07.257040    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" path="/var/lib/kubelet/pods/0afe197e-a5d9-406c-b52d-5c2b707a6857/volumes"
Nov 02 20:55:07.257485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:07.257438    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" path="/var/lib/kubelet/pods/43b4a54e-344e-4bf2-ab08-48b0724a4242/volumes"
Nov 02 20:55:09.813807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:09.813440    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:55:09.813807 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:09.813520    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:55:10.250806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:10.233158    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vgn28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:55:10.250806 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:10.233215    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vgn28" podUID="d1fd74f9-e682-4a78-b870-00402c71d4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:55:10.418808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:10.417975    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jskmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:55:10.418808 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:10.418046    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jskmn" podUID="fda36b26-75e5-45af-8fa1-ec906d88185b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:55:16.094798 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:55:16.094742    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:55:28.040894 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.040854    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 20:55:28.041447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.041097    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" podUID="94c3ac23-06e8-4666-aa36-3b2a842f0305" containerName="manager" containerID="cri-o://d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745" gracePeriod=10
Nov 02 20:55:28.309128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.309104    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 20:55:28.392009 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.391969    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\" (UniqueName: \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\") pod \"94c3ac23-06e8-4666-aa36-3b2a842f0305\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") "
Nov 02 20:55:28.392213 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.392055    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\") pod \"94c3ac23-06e8-4666-aa36-3b2a842f0305\" (UID: \"94c3ac23-06e8-4666-aa36-3b2a842f0305\") "
Nov 02 20:55:28.392485 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.392457    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume" pod "94c3ac23-06e8-4666-aa36-3b2a842f0305" (UID: "94c3ac23-06e8-4666-aa36-3b2a842f0305"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 20:55:28.394185 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.394163    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q" pod "94c3ac23-06e8-4666-aa36-3b2a842f0305" (UID: "94c3ac23-06e8-4666-aa36-3b2a842f0305"). InnerVolumeSpecName "kube-api-access-npz5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:55:28.492878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.492839    5099 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c3ac23-06e8-4666-aa36-3b2a842f0305-extensions-socket-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:28.492878 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.492869    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz5q\" (UniqueName: \"kubernetes.io/projected/94c3ac23-06e8-4666-aa36-3b2a842f0305-kube-api-access-npz5q\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:55:28.588941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.588848    5099 generic.go:356] "Generic (PLEG): container finished" podID="94c3ac23-06e8-4666-aa36-3b2a842f0305" containerID="d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745" exitCode=0
Nov 02 20:55:28.588941 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.588896    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"
Nov 02 20:55:28.589126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.588935    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" event={"ID":"94c3ac23-06e8-4666-aa36-3b2a842f0305","Type":"ContainerDied","Data":"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"}
Nov 02 20:55:28.589126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.588996    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb" event={"ID":"94c3ac23-06e8-4666-aa36-3b2a842f0305","Type":"ContainerDied","Data":"a7d790a59cf5639eac168b53bb18fa7cbb9879bc7f30cb25307c8ef802c8d8c2"}
Nov 02 20:55:28.589126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.589019    5099 scope.go:122] "RemoveContainer" containerID="d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"
Nov 02 20:55:28.599432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.599414    5099 scope.go:122] "RemoveContainer" containerID="d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"
Nov 02 20:55:28.599672 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:55:28.599654    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745\": container with ID starting with d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745 not found: ID does not exist" containerID="d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"
Nov 02 20:55:28.599744 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.599678    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745"} err="failed to get container status \"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745\": rpc error: code = NotFound desc = could not find container \"d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745\": container with ID starting with d98f99ba28548e9bf95e1cce77db5048fcdd22218032941d8b992db89e8b1745 not found: ID does not exist"
Nov 02 20:55:28.744042 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.744008    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 20:55:28.748093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:28.748068    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rllqb"]
Nov 02 20:55:29.256593 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:29.256557    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c3ac23-06e8-4666-aa36-3b2a842f0305" path="/var/lib/kubelet/pods/94c3ac23-06e8-4666-aa36-3b2a842f0305/volumes"
Nov 02 20:55:46.093457 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:55:46.093378    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:55:56.663409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:56.663376    5099 scope.go:122] "RemoveContainer" containerID="bed6293f5af4b593913b419abba3c282641f410a009fc24922f5f18120781ba6"
Nov 02 20:55:56.672101 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:55:56.672081    5099 scope.go:122] "RemoveContainer" containerID="406ee60144ab5e855fbccc91e477c2956f2ce5508af17e672141b32160090d4b"
Nov 02 20:56:16.093643 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:56:16.093599    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:56:31.622729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.622686    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"]
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623127    5099 cpu_manager.go:414] "containerMap: removing container" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623139    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623157    5099 cpu_manager.go:414] "containerMap: removing container" podUID="94c3ac23-06e8-4666-aa36-3b2a842f0305" containerName="manager"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623162    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c3ac23-06e8-4666-aa36-3b2a842f0305" containerName="manager"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623174    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623179    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623192    5099 cpu_manager.go:414] "containerMap: removing container" podUID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" containerName="maas-api"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623197    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" containerName="maas-api"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623260    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b3a1df-e741-495a-9cfa-d4e511fbd841" containerName="maas-api"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623273    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b4a54e-344e-4bf2-ab08-48b0724a4242" containerName="payload-processing"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623279    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c3ac23-06e8-4666-aa36-3b2a842f0305" containerName="manager"
Nov 02 20:56:31.623291 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.623285    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afe197e-a5d9-406c-b52d-5c2b707a6857" containerName="payload-pre-processing"
Nov 02 20:56:31.626524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.626507    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.675476 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.675443    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"]
Nov 02 20:56:31.719774 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.719741    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7xhk4\""
Nov 02 20:56:31.725215 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.725183    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spp95\" (UniqueName: \"kubernetes.io/projected/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-kube-api-access-spp95\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.725352 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.725247    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.825959 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.825921    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spp95\" (UniqueName: \"kubernetes.io/projected/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-kube-api-access-spp95\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.826153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.825975    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.826413 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.826392    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.883192 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.883114    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spp95\" (UniqueName: \"kubernetes.io/projected/ed5c8ea0-5f41-4610-bc9e-57e9d389ca10-kube-api-access-spp95\") pod \"kuadrant-operator-controller-manager-678d87766f-lldds\" (UID: \"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:31.939325 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:31.939296    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:32.352010 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:32.351983    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"]
Nov 02 20:56:32.353543 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:56:32.353518    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5c8ea0_5f41_4610_bc9e_57e9d389ca10.slice/crio-1f472d5fffa921b04bf6773dc661dcc1985519248194a9d69762d75a5e00aa75 WatchSource:0}: Error finding container 1f472d5fffa921b04bf6773dc661dcc1985519248194a9d69762d75a5e00aa75: Status 404 returned error can't find the container with id 1f472d5fffa921b04bf6773dc661dcc1985519248194a9d69762d75a5e00aa75
Nov 02 20:56:32.924663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:32.924620    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds" event={"ID":"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10","Type":"ContainerStarted","Data":"106e1965831b1545a0eba77970fdf47252ef15ab13eb440874ff580756eb92e5"}
Nov 02 20:56:32.925105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:32.924715    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds" event={"ID":"ed5c8ea0-5f41-4610-bc9e-57e9d389ca10","Type":"ContainerStarted","Data":"1f472d5fffa921b04bf6773dc661dcc1985519248194a9d69762d75a5e00aa75"}
Nov 02 20:56:32.925105 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:32.924766    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:33.088678 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:33.088627    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds" podStartSLOduration=2.088615356 podStartE2EDuration="2.088615356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:56:33.087418529 +0000 UTC m=+3881.961689419" watchObservedRunningTime="2026-11-02 20:56:33.088615356 +0000 UTC m=+3881.962886244"
Nov 02 20:56:43.932076 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:43.932044    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lldds"
Nov 02 20:56:44.051753 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:44.051719    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"]
Nov 02 20:56:44.063349 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:44.063321    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bt7qw"]
Nov 02 20:56:45.256632 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:45.256600    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9201be22-a76b-46dd-823c-ed174e80b366" path="/var/lib/kubelet/pods/9201be22-a76b-46dd-823c-ed174e80b366/volumes"
Nov 02 20:56:46.093615 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:56:46.093571    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:56:52.088600 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:52.088576    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:56:52.134177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:52.134151    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 20:56:54.076818 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:54.062708    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"]
Nov 02 20:56:54.136986 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:54.136936    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8t9sj"]
Nov 02 20:56:54.814028 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:54.813990    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:56:54.814364 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:54.814317    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:56:55.255887 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:55.255851    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea9366b-0dda-475f-82ce-6f23c15d025f" path="/var/lib/kubelet/pods/5ea9366b-0dda-475f-82ce-6f23c15d025f/volumes"
Nov 02 20:56:56.150944 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.150909    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"]
Nov 02 20:56:56.154558 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.154541    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.207129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.207090    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:56:56.209743 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.209715    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"]
Nov 02 20:56:56.246379 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.246351    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2gn\" (UniqueName: \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.246556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.246477    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.246556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.246519    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.254278 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.254249    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:56:56.347487 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.347453    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2gn\" (UniqueName: \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.348031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.347571    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.348031 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.347601    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.348229 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.348206    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.350158 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.350131    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.401259 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.401189    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2gn\" (UniqueName: \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\") pod \"maas-api-revoke-sub-d576e68b-zh4sh\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.474013 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.473978    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:56.709251 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.709169    5099 scope.go:122] "RemoveContainer" containerID="82636e36f26659196604dea611105368f65e5eb8a17afe2532d59db9024b78a4"
Nov 02 20:56:56.718118 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.718096    5099 scope.go:122] "RemoveContainer" containerID="46b438765214f4b1b433b727a8184638f139b10d47a5f1cd34be48ea688114f5"
Nov 02 20:56:56.839027 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:56.839002    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"]
Nov 02 20:56:56.840998 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:56:56.840968    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d560a71_9d49_4470_bff9_f9b139d11598.slice/crio-507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f WatchSource:0}: Error finding container 507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f: Status 404 returned error can't find the container with id 507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f
Nov 02 20:56:57.080690 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:57.080658    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh" event={"ID":"2d560a71-9d49-4470-bff9-f9b139d11598","Type":"ContainerStarted","Data":"507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f"}
Nov 02 20:56:58.087126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:58.087038    5099 generic.go:356] "Generic (PLEG): container finished" podID="2d560a71-9d49-4470-bff9-f9b139d11598" containerID="05bee947228d064c63a345e3fbcd57f38960369aac253547281013e5997237a7" exitCode=0
Nov 02 20:56:58.087126 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:58.087113    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh" event={"ID":"2d560a71-9d49-4470-bff9-f9b139d11598","Type":"ContainerDied","Data":"05bee947228d064c63a345e3fbcd57f38960369aac253547281013e5997237a7"}
Nov 02 20:56:59.243310 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.243278    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:56:59.276532 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.276500    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\") pod \"2d560a71-9d49-4470-bff9-f9b139d11598\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") "
Nov 02 20:56:59.276694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.276540    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\" (UniqueName: \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\") pod \"2d560a71-9d49-4470-bff9-f9b139d11598\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") "
Nov 02 20:56:59.276694 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.276643    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\") pod \"2d560a71-9d49-4470-bff9-f9b139d11598\" (UID: \"2d560a71-9d49-4470-bff9-f9b139d11598\") "
Nov 02 20:56:59.277056 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.277028    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca" pod "2d560a71-9d49-4470-bff9-f9b139d11598" (UID: "2d560a71-9d49-4470-bff9-f9b139d11598"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:56:59.278681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.278659    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn" pod "2d560a71-9d49-4470-bff9-f9b139d11598" (UID: "2d560a71-9d49-4470-bff9-f9b139d11598"). InnerVolumeSpecName "kube-api-access-xx2gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:56:59.278681 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.278672    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca" pod "2d560a71-9d49-4470-bff9-f9b139d11598" (UID: "2d560a71-9d49-4470-bff9-f9b139d11598"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:56:59.377542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.377499    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:59.377542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.377534    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2gn\" (UniqueName: \"kubernetes.io/projected/2d560a71-9d49-4470-bff9-f9b139d11598-kube-api-access-xx2gn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:56:59.377542 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:56:59.377545    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d560a71-9d49-4470-bff9-f9b139d11598-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:57:00.100627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:00.100590    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh" event={"ID":"2d560a71-9d49-4470-bff9-f9b139d11598","Type":"ContainerDied","Data":"507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f"}
Nov 02 20:57:00.100627 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:00.100612    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"
Nov 02 20:57:00.100913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:00.100642    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507b87b6a2407ac3bab7ecbcf86213f36accb054d40935026681923e21bbda8f"
Nov 02 20:57:16.267807 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:57:16.253166    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:57:25.053445 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.053404    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"]
Nov 02 20:57:25.054178 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.054154    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2d560a71-9d49-4470-bff9-f9b139d11598" containerName="revoke-keys"
Nov 02 20:57:25.054256 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.054188    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d560a71-9d49-4470-bff9-f9b139d11598" containerName="revoke-keys"
Nov 02 20:57:25.054330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.054318    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d560a71-9d49-4470-bff9-f9b139d11598" containerName="revoke-keys"
Nov 02 20:57:25.057886 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.057861    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.102686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.102648    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7hp\" (UniqueName: \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.102686 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.102700    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.103016 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.102778    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.109021 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.108984    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"]
Nov 02 20:57:25.152095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.152066    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 20:57:25.152248 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.152119    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 20:57:25.203432 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.203399    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7hp\" (UniqueName: \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.203650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.203437    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.203650 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.203459    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.204329 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.204301    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.205702 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.205679    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.257136 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.257107    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7hp\" (UniqueName: \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\") pod \"maas-api-revoke-sub-d2bc93bc-4c7z7\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.374372 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.374346    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:25.736112 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:25.736083    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"]
Nov 02 20:57:25.738074 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:57:25.738050    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd56b4d6_8b23_47af_8698_2ed5132825f0.slice/crio-7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade WatchSource:0}: Error finding container 7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade: Status 404 returned error can't find the container with id 7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade
Nov 02 20:57:26.290258 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:26.290225    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7" event={"ID":"fd56b4d6-8b23-47af-8698-2ed5132825f0","Type":"ContainerStarted","Data":"6b69e2412e595681a5d32294f35ca99e5ec7d12abe9fd8c1417ef03d3cc133b3"}
Nov 02 20:57:26.290655 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:26.290274    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7" event={"ID":"fd56b4d6-8b23-47af-8698-2ed5132825f0","Type":"ContainerStarted","Data":"7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade"}
Nov 02 20:57:26.398873 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:26.398820    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7" podStartSLOduration=1.398806784 podStartE2EDuration="1.398806784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:26.396951221 +0000 UTC m=+3935.271222109" watchObservedRunningTime="2026-11-02 20:57:26.398806784 +0000 UTC m=+3935.273077671"
Nov 02 20:57:27.297128 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:27.297087    5099 generic.go:356] "Generic (PLEG): container finished" podID="fd56b4d6-8b23-47af-8698-2ed5132825f0" containerID="6b69e2412e595681a5d32294f35ca99e5ec7d12abe9fd8c1417ef03d3cc133b3" exitCode=0
Nov 02 20:57:27.297598 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:27.297134    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7" event={"ID":"fd56b4d6-8b23-47af-8698-2ed5132825f0","Type":"ContainerDied","Data":"6b69e2412e595681a5d32294f35ca99e5ec7d12abe9fd8c1417ef03d3cc133b3"}
Nov 02 20:57:28.439524 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.439499    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:28.536676 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.536640    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\") pod \"fd56b4d6-8b23-47af-8698-2ed5132825f0\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") "
Nov 02 20:57:28.536895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.536745    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\" (UniqueName: \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\") pod \"fd56b4d6-8b23-47af-8698-2ed5132825f0\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") "
Nov 02 20:57:28.536895 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.536821    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\") pod \"fd56b4d6-8b23-47af-8698-2ed5132825f0\" (UID: \"fd56b4d6-8b23-47af-8698-2ed5132825f0\") "
Nov 02 20:57:28.537083 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.537047    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca" pod "fd56b4d6-8b23-47af-8698-2ed5132825f0" (UID: "fd56b4d6-8b23-47af-8698-2ed5132825f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:57:28.537187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.537166    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:57:28.538812 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.538777    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca" pod "fd56b4d6-8b23-47af-8698-2ed5132825f0" (UID: "fd56b4d6-8b23-47af-8698-2ed5132825f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:57:28.538916 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.538897    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp" pod "fd56b4d6-8b23-47af-8698-2ed5132825f0" (UID: "fd56b4d6-8b23-47af-8698-2ed5132825f0"). InnerVolumeSpecName "kube-api-access-mj7hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:57:28.637929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.637892    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7hp\" (UniqueName: \"kubernetes.io/projected/fd56b4d6-8b23-47af-8698-2ed5132825f0-kube-api-access-mj7hp\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:57:28.637929 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:28.637922    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd56b4d6-8b23-47af-8698-2ed5132825f0-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:57:29.309866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:29.309830    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"
Nov 02 20:57:29.309866 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:29.309844    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7" event={"ID":"fd56b4d6-8b23-47af-8698-2ed5132825f0","Type":"ContainerDied","Data":"7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade"}
Nov 02 20:57:29.310125 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:29.309893    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f413de5e982dedb1539334837407053da1d9de550ea53b865cedcaa07569ade"
Nov 02 20:57:46.093078 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:57:46.093039    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:57:46.292458 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.292420    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:57:46.292913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.292899    5099 cpu_manager.go:414] "containerMap: removing container" podUID="fd56b4d6-8b23-47af-8698-2ed5132825f0" containerName="revoke-keys"
Nov 02 20:57:46.292913 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.292914    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd56b4d6-8b23-47af-8698-2ed5132825f0" containerName="revoke-keys"
Nov 02 20:57:46.293030 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.292992    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd56b4d6-8b23-47af-8698-2ed5132825f0" containerName="revoke-keys"
Nov 02 20:57:46.297534 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.297510    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.347989 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.347899    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:57:46.393070 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.393034    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d512fb\""
Nov 02 20:57:46.393276 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.393225    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7802ee55-dockercfg-skgh8\""
Nov 02 20:57:46.430430 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.430385    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.430646 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.430587    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn78r\" (UniqueName: \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.436121 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.436090    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:57:46.439945 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.439923    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.492547 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.492512    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:57:46.531451 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.531418    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn78r\" (UniqueName: \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.531620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.531476    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5fr\" (UniqueName: \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.531620 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.531556    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.531703 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.531627    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.532321 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.532302    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.584960 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.584927    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn78r\" (UniqueName: \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\") pod \"payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.611048 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.610992    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:46.632556 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.632527    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5fr\" (UniqueName: \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.632705 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.632636    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.633549 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.633527    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.690055 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.690022    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5fr\" (UniqueName: \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\") pod \"payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.754515 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.754478    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:46.977237 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:46.977206    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:57:46.983676 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:57:46.983644    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5d42f7_d058_484a_9b3e_3202f497aeb1.slice/crio-0ee62bda53bc50107517c7bd22090ba79c6c5961081ac6e281b9319b1f28c2b0 WatchSource:0}: Error finding container 0ee62bda53bc50107517c7bd22090ba79c6c5961081ac6e281b9319b1f28c2b0: Status 404 returned error can't find the container with id 0ee62bda53bc50107517c7bd22090ba79c6c5961081ac6e281b9319b1f28c2b0
Nov 02 20:57:47.124693 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.124669    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:57:47.126588 ip-10-0-133-26 kubenswrapper[5099]: W1102 20:57:47.126527    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ebbc94_5230_492a_8df2_d13c19a46c2d.slice/crio-156fbcc3c07aac777f709431c88f991ded9abf66f40870b19c6fa9af19546d5e WatchSource:0}: Error finding container 156fbcc3c07aac777f709431c88f991ded9abf66f40870b19c6fa9af19546d5e: Status 404 returned error can't find the container with id 156fbcc3c07aac777f709431c88f991ded9abf66f40870b19c6fa9af19546d5e
Nov 02 20:57:47.418546 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.418503    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" event={"ID":"c1ebbc94-5230-492a-8df2-d13c19a46c2d","Type":"ContainerStarted","Data":"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"}
Nov 02 20:57:47.418752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.418564    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" event={"ID":"c1ebbc94-5230-492a-8df2-d13c19a46c2d","Type":"ContainerStarted","Data":"156fbcc3c07aac777f709431c88f991ded9abf66f40870b19c6fa9af19546d5e"}
Nov 02 20:57:47.418752 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.418689    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:57:47.420114 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.420086    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" event={"ID":"7d5d42f7-d058-484a-9b3e-3202f497aeb1","Type":"ContainerStarted","Data":"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"}
Nov 02 20:57:47.420234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.420128    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" event={"ID":"7d5d42f7-d058-484a-9b3e-3202f497aeb1","Type":"ContainerStarted","Data":"0ee62bda53bc50107517c7bd22090ba79c6c5961081ac6e281b9319b1f28c2b0"}
Nov 02 20:57:47.420234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.420172    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:47.526626 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.526569    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" podStartSLOduration=1.526554795 podStartE2EDuration="1.526554795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:47.52562831 +0000 UTC m=+3956.399899199" watchObservedRunningTime="2026-11-02 20:57:47.526554795 +0000 UTC m=+3956.400825682"
Nov 02 20:57:47.634758 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:47.634685    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" podStartSLOduration=1.6346686780000002 podStartE2EDuration="1.634668678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:47.633458134 +0000 UTC m=+3956.507729024" watchObservedRunningTime="2026-11-02 20:57:47.634668678 +0000 UTC m=+3956.508939568"
Nov 02 20:57:48.479107 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.479076    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:57:48.482642 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.482621    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.579759 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.579726    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d512fb-serving-cert\""
Nov 02 20:57:48.579972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.579837    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\""
Nov 02 20:57:48.579972 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.579895    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 20:57:48.582978 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.582952    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:57:48.656859 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.656829    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.657044 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.656979    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9xz\" (UniqueName: \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.657093 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.657071    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.657153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.657136    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.758820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.758714    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9xz\" (UniqueName: \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.758820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.758762    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.758820 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.758809    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.759095 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.758857    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.759582 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.759553    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.761795 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.761761    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.761914 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.761804    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:48.812933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:48.812900    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9xz\" (UniqueName: \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\") pod \"maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:49.096933 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:49.096843    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:49.468224 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:49.463645    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:57:50.442741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:50.441922    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" event={"ID":"a94c4d2b-a1c5-4045-b56f-86d4648cd623","Type":"ContainerStarted","Data":"fb9b43bfd904565e11e6c540c6c63f701da9d02009cd85d0a483b47232356cc0"}
Nov 02 20:57:51.449015 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:51.448973    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" event={"ID":"a94c4d2b-a1c5-4045-b56f-86d4648cd623","Type":"ContainerStarted","Data":"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"}
Nov 02 20:57:51.449526 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:51.449049    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:51.557480 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:51.557425    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" podStartSLOduration=2.523054057 podStartE2EDuration="3.557411501s" totalImagesPullingTime="1.034357444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:51.556203627 +0000 UTC m=+3960.430474517" watchObservedRunningTime="2026-11-02 20:57:51.557411501 +0000 UTC m=+3960.431682389"
Nov 02 20:57:56.747816 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:56.747760    5099 scope.go:122] "RemoveContainer" containerID="123a57ba60fbe692a1b5524227a71cfbc55e2c2cec7306a5b1252968b7284c98"
Nov 02 20:57:56.757193 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:56.757166    5099 scope.go:122] "RemoveContainer" containerID="2133e28b837fdbd03ae731e7734fb98ffaa06a8c8f5348ef2110c3cd8070ece2"
Nov 02 20:57:57.460531 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:57.460501    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:57:58.428741 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:58.428711    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:57:58.429200 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:57:58.428773    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:58:16.093587 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:58:16.093551    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:58:46.094030 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:58:46.093951    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:58:56.789738 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:58:56.789704    5099 scope.go:122] "RemoveContainer" containerID="fa4e8b93f48db5ab8fbc5db920d4ed14cb358bf46ccc5664d7fcd90bc83e61e5"
Nov 02 20:59:16.093619 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:59:16.093579    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:59:29.166663 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:29.166627    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:59:29.214867 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:29.214837    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:59:29.215080 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:29.215002    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" podUID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" containerName="maas-api" containerID="cri-o://ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f" gracePeriod=30
Nov 02 20:59:29.275456 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:29.275420    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:59:30.478649 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.478623    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:59:30.548184 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.548151    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\") pod \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") "
Nov 02 20:59:30.548357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.548240    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\") pod \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") "
Nov 02 20:59:30.548357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.548266    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\") pod \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") "
Nov 02 20:59:30.548357 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.548308    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\" (UniqueName: \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\") pod \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\" (UID: \"a94c4d2b-a1c5-4045-b56f-86d4648cd623\") "
Nov 02 20:59:30.548641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.548616    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca" pod "a94c4d2b-a1c5-4045-b56f-86d4648cd623" (UID: "a94c4d2b-a1c5-4045-b56f-86d4648cd623"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:30.550472 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.550441    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs" pod "a94c4d2b-a1c5-4045-b56f-86d4648cd623" (UID: "a94c4d2b-a1c5-4045-b56f-86d4648cd623"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:59:30.550599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.550471    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls" pod "a94c4d2b-a1c5-4045-b56f-86d4648cd623" (UID: "a94c4d2b-a1c5-4045-b56f-86d4648cd623"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 20:59:30.550599 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.550539    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz" pod "a94c4d2b-a1c5-4045-b56f-86d4648cd623" (UID: "a94c4d2b-a1c5-4045-b56f-86d4648cd623"). InnerVolumeSpecName "kube-api-access-hs9xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:30.649473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.649435    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:30.649473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.649468    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c4d2b-a1c5-4045-b56f-86d4648cd623-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:30.649473 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.649477    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c4d2b-a1c5-4045-b56f-86d4648cd623-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:30.649734 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:30.649486    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9xz\" (UniqueName: \"kubernetes.io/projected/a94c4d2b-a1c5-4045-b56f-86d4648cd623-kube-api-access-hs9xz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:31.001997 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.001959    5099 generic.go:356] "Generic (PLEG): container finished" podID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" containerID="ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f" exitCode=0
Nov 02 20:59:31.002177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.002010    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"
Nov 02 20:59:31.002177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.002038    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" event={"ID":"a94c4d2b-a1c5-4045-b56f-86d4648cd623","Type":"ContainerDied","Data":"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"}
Nov 02 20:59:31.002177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.002094    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj" event={"ID":"a94c4d2b-a1c5-4045-b56f-86d4648cd623","Type":"ContainerDied","Data":"fb9b43bfd904565e11e6c540c6c63f701da9d02009cd85d0a483b47232356cc0"}
Nov 02 20:59:31.002177 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.002115    5099 scope.go:122] "RemoveContainer" containerID="ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"
Nov 02 20:59:31.011641 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.011615    5099 scope.go:122] "RemoveContainer" containerID="ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"
Nov 02 20:59:31.011956 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:59:31.011936    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f\": container with ID starting with ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f not found: ID does not exist" containerID="ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"
Nov 02 20:59:31.012039 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.011963    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f"} err="failed to get container status \"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f\": rpc error: code = NotFound desc = could not find container \"ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f\": container with ID starting with ba0ed4ae1eea22a441b9c1216f3dc7a9c90c956b2e78b7a5c3b3a8624e141b5f not found: ID does not exist"
Nov 02 20:59:31.157723 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.157687    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:59:31.207330 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.207289    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d512fb-7db875555d-n97lj"]
Nov 02 20:59:31.256729 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:31.256655    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" path="/var/lib/kubelet/pods/a94c4d2b-a1c5-4045-b56f-86d4648cd623/volumes"
Nov 02 20:59:34.167815 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:34.167757    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing" containerID="cri-o://29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4" gracePeriod=25
Nov 02 20:59:34.276384 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:34.276325    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing" containerID="cri-o://9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9" gracePeriod=25
Nov 02 20:59:38.427374 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:38.427331    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:59:38.427915 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:38.427340    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:59:46.093857 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:59:46.093824    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 20:59:48.427799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:48.427749    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:59:48.427799 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:48.427777    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 20:59:54.870092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.849859    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 20:59:54.870092 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.849919    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 20:59:54.876220 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.876187    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:59:54.941018 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.940990    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:59:54.963982 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.963952    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\") pod \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") "
Nov 02 20:59:54.964153 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.963991    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\" (UniqueName: \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\") pod \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\" (UID: \"c1ebbc94-5230-492a-8df2-d13c19a46c2d\") "
Nov 02 20:59:54.964409 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.964384    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume" pod "c1ebbc94-5230-492a-8df2-d13c19a46c2d" (UID: "c1ebbc94-5230-492a-8df2-d13c19a46c2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:54.966082 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:54.966065    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr" pod "c1ebbc94-5230-492a-8df2-d13c19a46c2d" (UID: "c1ebbc94-5230-492a-8df2-d13c19a46c2d"). InnerVolumeSpecName "kube-api-access-6h5fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:55.065706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.065609    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\" (UniqueName: \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\") pod \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") "
Nov 02 20:59:55.065706 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.065700    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\") pod \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\" (UID: \"7d5d42f7-d058-484a-9b3e-3202f497aeb1\") "
Nov 02 20:59:55.066129 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.066103    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume" pod "7d5d42f7-d058-484a-9b3e-3202f497aeb1" (UID: "7d5d42f7-d058-484a-9b3e-3202f497aeb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 20:59:55.066234 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.066216    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d42f7-d058-484a-9b3e-3202f497aeb1-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:55.066293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.066237    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ebbc94-5230-492a-8df2-d13c19a46c2d-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:55.066293 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.066251    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5fr\" (UniqueName: \"kubernetes.io/projected/c1ebbc94-5230-492a-8df2-d13c19a46c2d-kube-api-access-6h5fr\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:55.067765 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.067744    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r" pod "7d5d42f7-d058-484a-9b3e-3202f497aeb1" (UID: "7d5d42f7-d058-484a-9b3e-3202f497aeb1"). InnerVolumeSpecName "kube-api-access-dn78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 20:59:55.167837 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.167803    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn78r\" (UniqueName: \"kubernetes.io/projected/7d5d42f7-d058-484a-9b3e-3202f497aeb1-kube-api-access-dn78r\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 20:59:55.244595 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.244561    5099 generic.go:356] "Generic (PLEG): container finished" podID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerID="29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4" exitCode=0
Nov 02 20:59:55.244813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.244624    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"
Nov 02 20:59:55.244813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.244648    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" event={"ID":"c1ebbc94-5230-492a-8df2-d13c19a46c2d","Type":"ContainerDied","Data":"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"}
Nov 02 20:59:55.244813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.244710    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48" event={"ID":"c1ebbc94-5230-492a-8df2-d13c19a46c2d","Type":"ContainerDied","Data":"156fbcc3c07aac777f709431c88f991ded9abf66f40870b19c6fa9af19546d5e"}
Nov 02 20:59:55.244813 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.244743    5099 scope.go:122] "RemoveContainer" containerID="29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"
Nov 02 20:59:55.246244 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.246218    5099 generic.go:356] "Generic (PLEG): container finished" podID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerID="9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9" exitCode=0
Nov 02 20:59:55.246378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.246280    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"
Nov 02 20:59:55.246378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.246303    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" event={"ID":"7d5d42f7-d058-484a-9b3e-3202f497aeb1","Type":"ContainerDied","Data":"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"}
Nov 02 20:59:55.246378 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.246365    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr" event={"ID":"7d5d42f7-d058-484a-9b3e-3202f497aeb1","Type":"ContainerDied","Data":"0ee62bda53bc50107517c7bd22090ba79c6c5961081ac6e281b9319b1f28c2b0"}
Nov 02 20:59:55.263315 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.263295    5099 scope.go:122] "RemoveContainer" containerID="29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"
Nov 02 20:59:55.263725 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:59:55.263694    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4\": container with ID starting with 29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4 not found: ID does not exist" containerID="29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"
Nov 02 20:59:55.263846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.263731    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4"} err="failed to get container status \"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4\": rpc error: code = NotFound desc = could not find container \"29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4\": container with ID starting with 29e368fcf9114b8a583226e00bda6136372dfbabea8d0a568c833821badb8ea4 not found: ID does not exist"
Nov 02 20:59:55.263846 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.263752    5099 scope.go:122] "RemoveContainer" containerID="9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"
Nov 02 20:59:55.272065 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.272046    5099 scope.go:122] "RemoveContainer" containerID="9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"
Nov 02 20:59:55.272351 ip-10-0-133-26 kubenswrapper[5099]: E1102 20:59:55.272326    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9\": container with ID starting with 9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9 not found: ID does not exist" containerID="9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"
Nov 02 20:59:55.272415 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.272361    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9"} err="failed to get container status \"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9\": rpc error: code = NotFound desc = could not find container \"9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9\": container with ID starting with 9907a9017f20c6e521a90476c597be33d79af60139b4a8df09c2eb95a3e288c9 not found: ID does not exist"
Nov 02 20:59:55.399447 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.399412    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:59:55.403187 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.403162    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d512fb-6f4f8s85pr"]
Nov 02 20:59:55.505955 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.505925    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:59:55.509331 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:55.509306    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d512fb-668677cd5hln48"]
Nov 02 20:59:57.256362 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:57.256330    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" path="/var/lib/kubelet/pods/7d5d42f7-d058-484a-9b3e-3202f497aeb1/volumes"
Nov 02 20:59:57.256792 ip-10-0-133-26 kubenswrapper[5099]: I1102 20:59:57.256765    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" path="/var/lib/kubelet/pods/c1ebbc94-5230-492a-8df2-d13c19a46c2d/volumes"
Nov 02 21:00:00.159824 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.159713    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160171    5099 cpu_manager.go:414] "containerMap: removing container" podUID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" containerName="maas-api"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160182    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" containerName="maas-api"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160209    5099 cpu_manager.go:414] "containerMap: removing container" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160214    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160223    5099 cpu_manager.go:414] "containerMap: removing container" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160228    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160301    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94c4d2b-a1c5-4045-b56f-86d4648cd623" containerName="maas-api"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160310    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ebbc94-5230-492a-8df2-d13c19a46c2d" containerName="payload-processing"
Nov 02 21:00:00.160319 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.160316    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5d42f7-d058-484a-9b3e-3202f497aeb1" containerName="payload-pre-processing"
Nov 02 21:00:00.165031 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.165010    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.215530 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.215488    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:00.259866 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.259836    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 21:00:00.330223 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.330172    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsph\" (UniqueName: \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.330424 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.330334    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.431153 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.431067    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.431153 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.431144    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsph\" (UniqueName: \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.431938 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.431912    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.484948 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.484915    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsph\" (UniqueName: \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\") pod \"maas-api-key-cleanup-29894220-5d4qg\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:00.777912 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:00.777814    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:01.138734 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:01.138705    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:01.140531 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:00:01.140502    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4ac808_9b7e_4cbc_95b4_f3c3336756ab.slice/crio-d0a89378aa8d99972297416e4443f2f548bbf7116cc45642fb145f6b3ba6be08 WatchSource:0}: Error finding container d0a89378aa8d99972297416e4443f2f548bbf7116cc45642fb145f6b3ba6be08: Status 404 returned error can't find the container with id d0a89378aa8d99972297416e4443f2f548bbf7116cc45642fb145f6b3ba6be08
Nov 02 21:00:01.278391 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:01.278357    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" event={"ID":"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab","Type":"ContainerStarted","Data":"d0a89378aa8d99972297416e4443f2f548bbf7116cc45642fb145f6b3ba6be08"}
Nov 02 21:00:02.284467 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:02.284431    5099 generic.go:356] "Generic (PLEG): container finished" podID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerID="b3ecee9d9f6c3abe95bc2bf4071f1d3a1f15cc2e9af25b0a21de63455064e5e4" exitCode=60
Nov 02 21:00:02.284907 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:02.284490    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" event={"ID":"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab","Type":"ContainerDied","Data":"b3ecee9d9f6c3abe95bc2bf4071f1d3a1f15cc2e9af25b0a21de63455064e5e4"}
Nov 02 21:00:02.284907 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:02.284714    5099 scope.go:122] "RemoveContainer" containerID="b3ecee9d9f6c3abe95bc2bf4071f1d3a1f15cc2e9af25b0a21de63455064e5e4"
Nov 02 21:00:03.291838 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:03.291798    5099 generic.go:356] "Generic (PLEG): container finished" podID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerID="3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a" exitCode=60
Nov 02 21:00:03.292388 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:03.291895    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" event={"ID":"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab","Type":"ContainerDied","Data":"3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a"}
Nov 02 21:00:03.292388 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:03.291961    5099 scope.go:122] "RemoveContainer" containerID="b3ecee9d9f6c3abe95bc2bf4071f1d3a1f15cc2e9af25b0a21de63455064e5e4"
Nov 02 21:00:03.292388 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:03.292127    5099 scope.go:122] "RemoveContainer" containerID="3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a"
Nov 02 21:00:03.292561 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:00:03.292388    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-5d4qg_odh-ai-gateway-infra(6a4ac808-9b7e-4cbc-95b4-f3c3336756ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab"
Nov 02 21:00:04.299013 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:04.298981    5099 scope.go:122] "RemoveContainer" containerID="3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a"
Nov 02 21:00:04.299411 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:00:04.299211    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-5d4qg_odh-ai-gateway-infra(6a4ac808-9b7e-4cbc-95b4-f3c3336756ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab"
Nov 02 21:00:15.250368 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:15.250337    5099 scope.go:122] "RemoveContainer" containerID="3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a"
Nov 02 21:00:16.093000 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:00:16.092963    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:00:16.361813 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:16.361767    5099 generic.go:356] "Generic (PLEG): container finished" podID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerID="cdfbd78853b7d59ce15eb53409ef8f524932023c21e87179c9ad94707df17be8" exitCode=60
Nov 02 21:00:16.362234 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:16.361839    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" event={"ID":"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab","Type":"ContainerDied","Data":"cdfbd78853b7d59ce15eb53409ef8f524932023c21e87179c9ad94707df17be8"}
Nov 02 21:00:16.362234 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:16.361896    5099 scope.go:122] "RemoveContainer" containerID="3d1db38213ac72464aafadbfdc8ae4c5ef2e5334fbe5c33bf3f8c0422940ff6a"
Nov 02 21:00:16.362234 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:16.362195    5099 scope.go:122] "RemoveContainer" containerID="cdfbd78853b7d59ce15eb53409ef8f524932023c21e87179c9ad94707df17be8"
Nov 02 21:00:16.362481 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:00:16.362457    5099 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-5d4qg_odh-ai-gateway-infra(6a4ac808-9b7e-4cbc-95b4-f3c3336756ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab"
Nov 02 21:00:17.478316 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.478285    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:17.621904 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.621880    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:17.699031 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.699001    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\") pod \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") "
Nov 02 21:00:17.699031 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.699035    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\" (UniqueName: \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\") pod \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\" (UID: \"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab\") "
Nov 02 21:00:17.699377 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.699355    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca" pod "6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" (UID: "6a4ac808-9b7e-4cbc-95b4-f3c3336756ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:00:17.701071 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.701053    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph" pod "6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" (UID: "6a4ac808-9b7e-4cbc-95b4-f3c3336756ab"). InnerVolumeSpecName "kube-api-access-6qsph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:00:17.800357 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.800273    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-maas-api-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:17.800357 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:17.800303    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qsph\" (UniqueName: \"kubernetes.io/projected/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab-kube-api-access-6qsph\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:00:18.375647 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:18.375615    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg" event={"ID":"6a4ac808-9b7e-4cbc-95b4-f3c3336756ab","Type":"ContainerDied","Data":"d0a89378aa8d99972297416e4443f2f548bbf7116cc45642fb145f6b3ba6be08"}
Nov 02 21:00:18.375861 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:18.375670    5099 scope.go:122] "RemoveContainer" containerID="cdfbd78853b7d59ce15eb53409ef8f524932023c21e87179c9ad94707df17be8"
Nov 02 21:00:18.375861 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:18.375628    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"
Nov 02 21:00:18.530459 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:18.530423    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:18.577564 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:18.577532    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-5d4qg"]
Nov 02 21:00:19.256261 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:19.256228    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" path="/var/lib/kubelet/pods/6a4ac808-9b7e-4cbc-95b4-f3c3336756ab/volumes"
Nov 02 21:00:46.093589 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:00:46.093550    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:00:59.037847 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.037805    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"]
Nov 02 21:00:59.038428 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038410    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.038474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038429    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.038474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038468    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.038474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038473    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.038579 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038541    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.038579 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.038554    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:00:59.042120 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.042095    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.089708 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.089675    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4c8c\" (UniqueName: \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.089888 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.089714    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.089888 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.089758    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.094159 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.094132    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"]
Nov 02 21:00:59.138270 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.138236    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:00:59.138431 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.138395    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 21:00:59.191312 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.191277    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4c8c\" (UniqueName: \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.191312 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.191317    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.191545 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.191360    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.192119 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.192096    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.193718 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.193700    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.245196 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.245167    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4c8c\" (UniqueName: \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\") pod \"maas-api-revoke-sub-452b6d7b-v8tqz\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.355372 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.355287    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:00:59.718286 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:00:59.718257    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"]
Nov 02 21:00:59.719692 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:00:59.719664    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c84be02_27c7_4ca5_8c1b_a9e42cd9e0cb.slice/crio-58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c WatchSource:0}: Error finding container 58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c: Status 404 returned error can't find the container with id 58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c
Nov 02 21:01:00.639071 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:00.639031    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz" event={"ID":"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb","Type":"ContainerStarted","Data":"cff9d65f355428cf94d420f7b9e7e3ae43a2edb9d0dfb563bc3ba2bc970815f6"}
Nov 02 21:01:00.639594 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:00.639099    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz" event={"ID":"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb","Type":"ContainerStarted","Data":"58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c"}
Nov 02 21:01:00.749969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:00.749890    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz" podStartSLOduration=2.749871909 podStartE2EDuration="2.749871909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:01:00.74965253 +0000 UTC m=+4149.623923443" watchObservedRunningTime="2026-11-02 21:01:00.749871909 +0000 UTC m=+4149.624142798"
Nov 02 21:01:01.649568 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:01.649527    5099 generic.go:356] "Generic (PLEG): container finished" podID="2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" containerID="cff9d65f355428cf94d420f7b9e7e3ae43a2edb9d0dfb563bc3ba2bc970815f6" exitCode=0
Nov 02 21:01:01.650147 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:01.649618    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz" event={"ID":"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb","Type":"ContainerDied","Data":"cff9d65f355428cf94d420f7b9e7e3ae43a2edb9d0dfb563bc3ba2bc970815f6"}
Nov 02 21:01:03.590819 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.587199    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:01:03.723733 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.723690    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz" event={"ID":"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb","Type":"ContainerDied","Data":"58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c"}
Nov 02 21:01:03.723969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.723758    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e57592d4cc5ef0026b2e438dc307342a302ed58fd4e7fc9fd9a1086389fe0c"
Nov 02 21:01:03.723969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.723857    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"
Nov 02 21:01:03.803866 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.792346    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\") pod \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") "
Nov 02 21:01:03.835807 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.809212    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\" (UniqueName: \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\") pod \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") "
Nov 02 21:01:03.837601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.837572    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\") pod \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\" (UID: \"2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb\") "
Nov 02 21:01:03.838559 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.809004    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca" pod "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" (UID: "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:01:03.881032 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.864997    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca" pod "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" (UID: "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:01:03.943004 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.942949    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c" pod "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" (UID: "2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb"). InnerVolumeSpecName "kube-api-access-r4c8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:01:03.943550 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.943520    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:03.943748 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.943726    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4c8c\" (UniqueName: \"kubernetes.io/projected/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-kube-api-access-r4c8c\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:03.943898 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:03.943882    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:16.108200 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:01:16.108147    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:01:24.730777 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.730723    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"]
Nov 02 21:01:24.732172 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732142    5099 cpu_manager.go:414] "containerMap: removing container" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:01:24.732172 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732172    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:01:24.732365 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732228    5099 cpu_manager.go:414] "containerMap: removing container" podUID="2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" containerName="revoke-keys"
Nov 02 21:01:24.732365 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732237    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" containerName="revoke-keys"
Nov 02 21:01:24.732365 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732361    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4ac808-9b7e-4cbc-95b4-f3c3336756ab" containerName="cleanup"
Nov 02 21:01:24.732578 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.732375    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" containerName="revoke-keys"
Nov 02 21:01:24.737111 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.737090    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.794097 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.794000    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"]
Nov 02 21:01:24.816732 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.816695    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.816732 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.816737    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.817080 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.816773    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2zn\" (UniqueName: \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.834281 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.834253    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 21:01:24.835286 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.835252    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:01:24.917883 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.917834    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.917883 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.917891    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.918204 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.917949    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2zn\" (UniqueName: \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.919138 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.919104    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.920578 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.920552    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:24.982063 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:24.981980    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2zn\" (UniqueName: \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\") pod \"maas-api-revoke-sub-3e31a506-85h4b\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:25.057728 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:25.057689    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:25.734142 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:25.734107    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"]
Nov 02 21:01:25.934493 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:25.934407    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b" event={"ID":"e6cf4eb4-04f1-414c-b04c-a722b405715a","Type":"ContainerStarted","Data":"2071abe7bb52eb08ba6a5b2c62a872626ecf5ad4f697fb711af0726f1fdb1060"}
Nov 02 21:01:26.977639 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:26.976734    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b" event={"ID":"e6cf4eb4-04f1-414c-b04c-a722b405715a","Type":"ContainerStarted","Data":"f33f2bf2e6807c4e17786ef0f52ce18c271c9d454a07985bc6ce9661e065b4ae"}
Nov 02 21:01:27.991284 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:27.991243    5099 generic.go:356] "Generic (PLEG): container finished" podID="e6cf4eb4-04f1-414c-b04c-a722b405715a" containerID="f33f2bf2e6807c4e17786ef0f52ce18c271c9d454a07985bc6ce9661e065b4ae" exitCode=0
Nov 02 21:01:27.991743 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:27.991315    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b" event={"ID":"e6cf4eb4-04f1-414c-b04c-a722b405715a","Type":"ContainerDied","Data":"f33f2bf2e6807c4e17786ef0f52ce18c271c9d454a07985bc6ce9661e065b4ae"}
Nov 02 21:01:29.135434 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.135410    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:29.279339 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.279251    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\") pod \"e6cf4eb4-04f1-414c-b04c-a722b405715a\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") "
Nov 02 21:01:29.279339 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.279295    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\") pod \"e6cf4eb4-04f1-414c-b04c-a722b405715a\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") "
Nov 02 21:01:29.279601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.279501    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\" (UniqueName: \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\") pod \"e6cf4eb4-04f1-414c-b04c-a722b405715a\" (UID: \"e6cf4eb4-04f1-414c-b04c-a722b405715a\") "
Nov 02 21:01:29.279691 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.279669    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca" pod "e6cf4eb4-04f1-414c-b04c-a722b405715a" (UID: "e6cf4eb4-04f1-414c-b04c-a722b405715a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:01:29.279837 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.279822    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:29.281445 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.281415    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca" pod "e6cf4eb4-04f1-414c-b04c-a722b405715a" (UID: "e6cf4eb4-04f1-414c-b04c-a722b405715a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:01:29.281604 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.281582    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn" pod "e6cf4eb4-04f1-414c-b04c-a722b405715a" (UID: "e6cf4eb4-04f1-414c-b04c-a722b405715a"). InnerVolumeSpecName "kube-api-access-xp2zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:01:29.381422 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.381377    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2zn\" (UniqueName: \"kubernetes.io/projected/e6cf4eb4-04f1-414c-b04c-a722b405715a-kube-api-access-xp2zn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:29.381422 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:29.381417    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cf4eb4-04f1-414c-b04c-a722b405715a-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:30.002969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:30.002879    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"
Nov 02 21:01:30.002969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:30.002906    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b" event={"ID":"e6cf4eb4-04f1-414c-b04c-a722b405715a","Type":"ContainerDied","Data":"2071abe7bb52eb08ba6a5b2c62a872626ecf5ad4f697fb711af0726f1fdb1060"}
Nov 02 21:01:30.002969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:30.002966    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2071abe7bb52eb08ba6a5b2c62a872626ecf5ad4f697fb711af0726f1fdb1060"
Nov 02 21:01:46.093760 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:01:46.093718    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:01:52.122519 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:52.122490    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 21:01:52.169773 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:52.169748    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 21:01:55.648726 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.648693    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"]
Nov 02 21:01:55.649165 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.649151    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e6cf4eb4-04f1-414c-b04c-a722b405715a" containerName="revoke-keys"
Nov 02 21:01:55.649165 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.649165    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cf4eb4-04f1-414c-b04c-a722b405715a" containerName="revoke-keys"
Nov 02 21:01:55.649257 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.649248    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cf4eb4-04f1-414c-b04c-a722b405715a" containerName="revoke-keys"
Nov 02 21:01:55.652831 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.652807    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.708258 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.707230    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"]
Nov 02 21:01:55.748762 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.748732    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 21:01:55.748951 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.748858    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\""
Nov 02 21:01:55.780317 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.780284    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.780494 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.780422    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8snn\" (UniqueName: \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.780494 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.780471    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.881436 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.881399    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8snn\" (UniqueName: \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.881616 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.881461    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.881616 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.881512    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.882289 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.882262    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.883922 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.883883    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.934657 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.934593    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8snn\" (UniqueName: \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\") pod \"maas-api-revoke-sub-12b1d91f-p2srs\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:55.970477 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:55.970444    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:56.333550 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:56.333519    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"]
Nov 02 21:01:56.334867 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:01:56.334838    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff4f8ee_9218_4e68_9112_152219e17004.slice/crio-3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3 WatchSource:0}: Error finding container 3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3: Status 404 returned error can't find the container with id 3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3
Nov 02 21:01:57.263478 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:57.263397    5099 generic.go:356] "Generic (PLEG): container finished" podID="eff4f8ee-9218-4e68-9112-152219e17004" containerID="8665628c56eb8e620ad1ca8eea0569348a2cd824d076cc0d139ad381de902b2f" exitCode=0
Nov 02 21:01:57.263478 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:57.263457    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs" event={"ID":"eff4f8ee-9218-4e68-9112-152219e17004","Type":"ContainerDied","Data":"8665628c56eb8e620ad1ca8eea0569348a2cd824d076cc0d139ad381de902b2f"}
Nov 02 21:01:57.263917 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:57.263491    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs" event={"ID":"eff4f8ee-9218-4e68-9112-152219e17004","Type":"ContainerStarted","Data":"3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3"}
Nov 02 21:01:58.402500 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.402475    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:58.509109 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.509031    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\") pod \"eff4f8ee-9218-4e68-9112-152219e17004\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") "
Nov 02 21:01:58.509656 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.509620    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca" pod "eff4f8ee-9218-4e68-9112-152219e17004" (UID: "eff4f8ee-9218-4e68-9112-152219e17004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:01:58.509825 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.509712    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\" (UniqueName: \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\") pod \"eff4f8ee-9218-4e68-9112-152219e17004\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") "
Nov 02 21:01:58.509825 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.509767    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\") pod \"eff4f8ee-9218-4e68-9112-152219e17004\" (UID: \"eff4f8ee-9218-4e68-9112-152219e17004\") "
Nov 02 21:01:58.511976 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.511943    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-openshift-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:58.512411 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.512366    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn" pod "eff4f8ee-9218-4e68-9112-152219e17004" (UID: "eff4f8ee-9218-4e68-9112-152219e17004"). InnerVolumeSpecName "kube-api-access-w8snn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:01:58.512565 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.512536    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca" pod "eff4f8ee-9218-4e68-9112-152219e17004" (UID: "eff4f8ee-9218-4e68-9112-152219e17004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:01:58.612826 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.612752    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8snn\" (UniqueName: \"kubernetes.io/projected/eff4f8ee-9218-4e68-9112-152219e17004-kube-api-access-w8snn\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:58.612826 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:58.612822    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff4f8ee-9218-4e68-9112-152219e17004-maas-api-xks-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:01:59.274636 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:59.274610    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"
Nov 02 21:01:59.274800 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:59.274650    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs" event={"ID":"eff4f8ee-9218-4e68-9112-152219e17004","Type":"ContainerDied","Data":"3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3"}
Nov 02 21:01:59.274800 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:01:59.274695    5099 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3343bb9a86ce48597250207558fdb54b21d0c26506e9c655f0875748c091b0d3"
Nov 02 21:02:00.046002 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:00.045964    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"]
Nov 02 21:02:00.093464 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:00.093430    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-zh4sh"]
Nov 02 21:02:01.256983 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:01.256946    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d560a71-9d49-4470-bff9-f9b139d11598" path="/var/lib/kubelet/pods/2d560a71-9d49-4470-bff9-f9b139d11598/volumes"
Nov 02 21:02:08.888123 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.888087    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"]
Nov 02 21:02:08.889347 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.889316    5099 cpu_manager.go:414] "containerMap: removing container" podUID="eff4f8ee-9218-4e68-9112-152219e17004" containerName="revoke-keys"
Nov 02 21:02:08.889489 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.889475    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff4f8ee-9218-4e68-9112-152219e17004" containerName="revoke-keys"
Nov 02 21:02:08.889704 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.889693    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff4f8ee-9218-4e68-9112-152219e17004" containerName="revoke-keys"
Nov 02 21:02:08.894541 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.894518    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:08.926021 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.925981    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-metrics-certs\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:08.927124 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.926660    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-maas-api-tls\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:08.927124 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.926705    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/2f631b71-7320-4b80-8f07-b0340438ac2e-kube-api-access-2w9rq\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:08.927124 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.926791    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f631b71-7320-4b80-8f07-b0340438ac2e-openshift-service-ca\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:08.943814 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.943760    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 21:02:08.991201 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.991147    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 21:02:08.991201 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.991277    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\""
Nov 02 21:02:08.995235 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:08.995200    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"]
Nov 02 21:02:09.027791 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.027753    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-metrics-certs\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.027985 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.027856    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-maas-api-tls\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.027985 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.027881    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/2f631b71-7320-4b80-8f07-b0340438ac2e-kube-api-access-2w9rq\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.027985 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.027937    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f631b71-7320-4b80-8f07-b0340438ac2e-openshift-service-ca\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.028662 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.028636    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f631b71-7320-4b80-8f07-b0340438ac2e-openshift-service-ca\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.030490 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.030467    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-metrics-certs\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.030576 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.030495    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f631b71-7320-4b80-8f07-b0340438ac2e-maas-api-tls\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.081722 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.081690    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/2f631b71-7320-4b80-8f07-b0340438ac2e-kube-api-access-2w9rq\") pod \"maas-api-5bc7c89d86-brd4t\" (UID: \"2f631b71-7320-4b80-8f07-b0340438ac2e\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.231609 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.231512    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:09.598383 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.598352    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"]
Nov 02 21:02:09.599858 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:02:09.599822    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f631b71_7320_4b80_8f07_b0340438ac2e.slice/crio-2fa5bcfafd3359c38f0a044d908a10b8aa6775fedfdab71d87792e77ec8dbdf8 WatchSource:0}: Error finding container 2fa5bcfafd3359c38f0a044d908a10b8aa6775fedfdab71d87792e77ec8dbdf8: Status 404 returned error can't find the container with id 2fa5bcfafd3359c38f0a044d908a10b8aa6775fedfdab71d87792e77ec8dbdf8
Nov 02 21:02:09.601178 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:09.601157    5099 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 21:02:10.341822 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:10.341776    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t" event={"ID":"2f631b71-7320-4b80-8f07-b0340438ac2e","Type":"ContainerStarted","Data":"2fa5bcfafd3359c38f0a044d908a10b8aa6775fedfdab71d87792e77ec8dbdf8"}
Nov 02 21:02:11.348916 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:11.348882    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t" event={"ID":"2f631b71-7320-4b80-8f07-b0340438ac2e","Type":"ContainerStarted","Data":"4c2ae6c4b8fa3bf62443d68f47686ac378269e7a9e8d382a8ced1313efbc044e"}
Nov 02 21:02:11.349351 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:11.348986    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:11.456360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:11.456313    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t" podStartSLOduration=2.510153014 podStartE2EDuration="3.456299655s" totalImagesPullingTime="946.146641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:11.455669193 +0000 UTC m=+4220.329940083" watchObservedRunningTime="2026-11-02 21:02:11.456299655 +0000 UTC m=+4220.330570542"
Nov 02 21:02:16.093334 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:02:16.093295    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:02:17.361360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:17.361330    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-brd4t"
Nov 02 21:02:26.033351 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.033315    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:02:26.037532 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.037513    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.097748 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.097706    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098160 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098121    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098299 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098192    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098299 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098222    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098299 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098258    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlmjd\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098318    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098345    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098387    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.098474 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.098430    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.109080 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.109053    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:02:26.148206 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.148013    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3203ddac-openshift-default-dockercfg-tx4l8\""
Nov 02 21:02:26.199824 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.199735    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.199824 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.199824    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.199878    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.199907    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.199942    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmjd\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.200010    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.200035    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.200086    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.200132    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.200687 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.200659    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.202399 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.201678    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.202399 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.201986    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.202399 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.202056    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.216996 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.202757    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.216996 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.211392    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.221100 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.221072    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.257413 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.257383    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.257678 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.257657    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlmjd\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\") pod \"e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") " pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.350908 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.350815    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:26.767570 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.767539    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:02:26.768608 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:02:26.768585    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d80abc_5d9f_4303_bc46_b3aa349a1eaf.slice/crio-5f95ca4c33cc4a7cb7136a7a1c4a0c7ee30c6fc169bccc3da3e95a667b03ef59 WatchSource:0}: Error finding container 5f95ca4c33cc4a7cb7136a7a1c4a0c7ee30c6fc169bccc3da3e95a667b03ef59: Status 404 returned error can't find the container with id 5f95ca4c33cc4a7cb7136a7a1c4a0c7ee30c6fc169bccc3da3e95a667b03ef59
Nov 02 21:02:26.770904 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.770870    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:26.771002 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.770934    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:26.771002 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:26.770963    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:27.435850 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:27.435809    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerStarted","Data":"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"}
Nov 02 21:02:27.436355 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:27.435870    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerStarted","Data":"5f95ca4c33cc4a7cb7136a7a1c4a0c7ee30c6fc169bccc3da3e95a667b03ef59"}
Nov 02 21:02:27.593376 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:27.593322    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podStartSLOduration=2.593307908 podStartE2EDuration="2.593307908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:27.59233057 +0000 UTC m=+4236.466601459" watchObservedRunningTime="2026-11-02 21:02:27.593307908 +0000 UTC m=+4236.467578797"
Nov 02 21:02:28.351553 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:28.351498    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:28.356607 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:28.356582    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:28.441627 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:28.441596    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:28.442447 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:28.442427    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:29.043733 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:29.043695    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"]
Nov 02 21:02:29.045975 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:29.045953    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4c7z7"]
Nov 02 21:02:29.256946 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:29.256914    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd56b4d6-8b23-47af-8698-2ed5132825f0" path="/var/lib/kubelet/pods/fd56b4d6-8b23-47af-8698-2ed5132825f0/volumes"
Nov 02 21:02:33.694817 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.694760    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:02:33.702164 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.702136    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.748230 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.748187    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:02:33.787834 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.787774    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.787996 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.787844    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhj5\" (UniqueName: \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.792404 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.792381    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3203ddac\""
Nov 02 21:02:33.792502 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.792441    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3203ddac-dockercfg-pcjp7\""
Nov 02 21:02:33.889367 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.889336    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhj5\" (UniqueName: \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.889584 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.889445    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.890128 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.890108    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:33.943429 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:33.943395    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhj5\" (UniqueName: \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\") pod \"payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:34.016548 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.016469    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:34.380932 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.380905    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:02:34.382587 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:02:34.382558    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d34aa2d_5803_4eac_bcd4_292da8d6968b.slice/crio-f7d71a93d747f978b644cd7e58847fb1822e83960d8a8a6f9f4b579144eb5dbe WatchSource:0}: Error finding container f7d71a93d747f978b644cd7e58847fb1822e83960d8a8a6f9f4b579144eb5dbe: Status 404 returned error can't find the container with id f7d71a93d747f978b644cd7e58847fb1822e83960d8a8a6f9f4b579144eb5dbe
Nov 02 21:02:34.476044 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.476015    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" event={"ID":"0d34aa2d-5803-4eac-bcd4-292da8d6968b","Type":"ContainerStarted","Data":"8a145b68d42c8aa9c97328c6a4304e810d720ed065f6238daefdfedec50cc861"}
Nov 02 21:02:34.476185 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.476068    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" event={"ID":"0d34aa2d-5803-4eac-bcd4-292da8d6968b","Type":"ContainerStarted","Data":"f7d71a93d747f978b644cd7e58847fb1822e83960d8a8a6f9f4b579144eb5dbe"}
Nov 02 21:02:34.476185 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.476102    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:34.584741 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:34.584693    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" podStartSLOduration=1.584679354 podStartE2EDuration="1.584679354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:34.583358131 +0000 UTC m=+4243.457629021" watchObservedRunningTime="2026-11-02 21:02:34.584679354 +0000 UTC m=+4243.458950241"
Nov 02 21:02:45.484998 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:45.484965    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:02:46.093996 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:02:46.093958    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:02:50.597195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.597090    5099 generic.go:356] "Generic (PLEG): container finished" podID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerID="1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff" exitCode=0
Nov 02 21:02:50.597195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.597164    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerDied","Data":"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"}
Nov 02 21:02:50.597829 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.597651    5099 scope.go:122] "RemoveContainer" containerID="1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"
Nov 02 21:02:50.598924 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.598893    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:50.599062 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.598970    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:50.599062 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:50.599017    5099 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 21:02:51.605052 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:51.605019    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerStarted","Data":"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"}
Nov 02 21:02:52.358808 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:52.357195    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:52.358808 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:52.357351    5099 patch_prober.go:29] interesting pod/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Nov 02 21:02:52.358808 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:52.357399    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Nov 02 21:02:53.357805 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:53.357249    5099 patch_prober.go:29] interesting pod/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Nov 02 21:02:53.357805 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:53.357323    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Nov 02 21:02:54.358809 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:54.353188    5099 patch_prober.go:29] interesting pod/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Nov 02 21:02:54.358809 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:54.353264    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Nov 02 21:02:55.357806 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:55.354223    5099 patch_prober.go:29] interesting pod/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Nov 02 21:02:55.357806 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:55.354289    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Nov 02 21:02:56.355805 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:56.355209    5099 patch_prober.go:29] interesting pod/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Nov 02 21:02:56.355805 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:56.355286    5099 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Nov 02 21:02:57.355436 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:57.355357    5099 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:57.355831 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:57.355814    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:02:57.356574 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:02:57.356552    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:03:16.094164 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:16.094075    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:03:22.673239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:22.673196    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:03:27.673706 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.673661    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" podUID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" containerName="payload-pre-processing" containerID="cri-o://8a145b68d42c8aa9c97328c6a4304e810d720ed065f6238daefdfedec50cc861" gracePeriod=25
Nov 02 21:03:27.813747 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.813714    5099 generic.go:356] "Generic (PLEG): container finished" podID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" containerID="8a145b68d42c8aa9c97328c6a4304e810d720ed065f6238daefdfedec50cc861" exitCode=0
Nov 02 21:03:27.813949 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.813807    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" event={"ID":"0d34aa2d-5803-4eac-bcd4-292da8d6968b","Type":"ContainerDied","Data":"8a145b68d42c8aa9c97328c6a4304e810d720ed065f6238daefdfedec50cc861"}
Nov 02 21:03:27.939281 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.939219    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:03:27.959403 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.959380    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\" (UniqueName: \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\") pod \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") "
Nov 02 21:03:27.959538 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.959471    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\") pod \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\" (UID: \"0d34aa2d-5803-4eac-bcd4-292da8d6968b\") "
Nov 02 21:03:27.959866 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.959835    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume" pod "0d34aa2d-5803-4eac-bcd4-292da8d6968b" (UID: "0d34aa2d-5803-4eac-bcd4-292da8d6968b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:03:27.961501 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:27.961475    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5" pod "0d34aa2d-5803-4eac-bcd4-292da8d6968b" (UID: "0d34aa2d-5803-4eac-bcd4-292da8d6968b"). InnerVolumeSpecName "kube-api-access-jbhj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:28.060488 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.060457    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhj5\" (UniqueName: \"kubernetes.io/projected/0d34aa2d-5803-4eac-bcd4-292da8d6968b-kube-api-access-jbhj5\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:28.060488 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.060486    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34aa2d-5803-4eac-bcd4-292da8d6968b-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:28.820730 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.820704    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"
Nov 02 21:03:28.821300 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.820727    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk" event={"ID":"0d34aa2d-5803-4eac-bcd4-292da8d6968b","Type":"ContainerDied","Data":"f7d71a93d747f978b644cd7e58847fb1822e83960d8a8a6f9f4b579144eb5dbe"}
Nov 02 21:03:28.821300 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.820809    5099 scope.go:122] "RemoveContainer" containerID="8a145b68d42c8aa9c97328c6a4304e810d720ed065f6238daefdfedec50cc861"
Nov 02 21:03:28.981991 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:28.981955    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:03:29.029295 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:29.029251    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3203ddac-749cfc6579-7rgxk"]
Nov 02 21:03:29.256637 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:29.256603    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" path="/var/lib/kubelet/pods/0d34aa2d-5803-4eac-bcd4-292da8d6968b/volumes"
Nov 02 21:03:32.133793 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:32.133759    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:03:32.134215 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:32.134000    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy" containerID="cri-o://dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a" gracePeriod=30
Nov 02 21:03:37.395351 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.395319    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:03:37.474601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474517    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474564    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474595    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474929 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474617    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474929 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474690    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474929 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474734    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.474929 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474855    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:37.474929 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474865    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.475195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.474956    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.475195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475036    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\") pod \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\" (UID: \"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf\") "
Nov 02 21:03:37.475195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475077    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:37.475195 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475088    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:37.475407 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475291    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:37.475502 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475478    5099 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.475561 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475509    5099 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-credential-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.475561 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475524    5099 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-data\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.475561 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475539    5099 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-workload-socket\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.475561 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.475484    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:03:37.477009 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.476977    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 21:03:37.477284 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.477251    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:37.477426 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.477412    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "kube-api-access-mlmjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:03:37.477544 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.477420    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo" pod "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" (UID: "e2d80abc-5d9f-4303-bc46-b3aa349a1eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 21:03:37.576291 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.576258    5099 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-podinfo\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.576291 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.576290    5099 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istiod-ca-cert\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.576506 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.576302    5099 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-envoy\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.576506 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.576315    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlmjd\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-kube-api-access-mlmjd\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.576506 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.576328    5099 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf-istio-token\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:03:37.877698 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.877658    5099 generic.go:356] "Generic (PLEG): container finished" podID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerID="dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a" exitCode=0
Nov 02 21:03:37.877915 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.877715    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"
Nov 02 21:03:37.877915 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.877749    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerDied","Data":"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"}
Nov 02 21:03:37.877915 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.877832    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb" event={"ID":"e2d80abc-5d9f-4303-bc46-b3aa349a1eaf","Type":"ContainerDied","Data":"5f95ca4c33cc4a7cb7136a7a1c4a0c7ee30c6fc169bccc3da3e95a667b03ef59"}
Nov 02 21:03:37.877915 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.877863    5099 scope.go:122] "RemoveContainer" containerID="dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"
Nov 02 21:03:37.888093 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.888070    5099 scope.go:122] "RemoveContainer" containerID="1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"
Nov 02 21:03:37.896333 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.896303    5099 scope.go:122] "RemoveContainer" containerID="dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"
Nov 02 21:03:37.896587 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:37.896568    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a\": container with ID starting with dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a not found: ID does not exist" containerID="dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"
Nov 02 21:03:37.896637 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.896597    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a"} err="failed to get container status \"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a\": rpc error: code = NotFound desc = could not find container \"dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a\": container with ID starting with dc481f65603f3cc6c0ea4bae7517978e292df5f826b6070279804d7a88e7558a not found: ID does not exist"
Nov 02 21:03:37.896637 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.896617    5099 scope.go:122] "RemoveContainer" containerID="1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"
Nov 02 21:03:37.896877 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:37.896858    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff\": container with ID starting with 1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff not found: ID does not exist" containerID="1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"
Nov 02 21:03:37.896930 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:37.896884    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff"} err="failed to get container status \"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff\": rpc error: code = NotFound desc = could not find container \"1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff\": container with ID starting with 1d47caf96e52ce019ff63acbc4369205a48d3b93593a90c2e09b6d6f1700d9ff not found: ID does not exist"
Nov 02 21:03:38.034410 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:38.034375    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:03:38.082259 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:38.082221    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3203ddac-openshift-default-64d4b7485-9p6wb"]
Nov 02 21:03:39.257372 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:39.257337    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" path="/var/lib/kubelet/pods/e2d80abc-5d9f-4303-bc46-b3aa349a1eaf/volumes"
Nov 02 21:03:41.180734 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.180693    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181149    5099 cpu_manager.go:414] "containerMap: removing container" podUID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" containerName="payload-pre-processing"
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181161    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" containerName="payload-pre-processing"
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181176    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181181    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181201    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.181239 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181206    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.181456 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181287    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d34aa2d-5803-4eac-bcd4-292da8d6968b" containerName="payload-pre-processing"
Nov 02 21:03:41.181456 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181297    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.181456 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.181307    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d80abc-5d9f-4303-bc46-b3aa349a1eaf" containerName="istio-proxy"
Nov 02 21:03:41.186297 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.186277    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.236606 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.236574    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:03:41.282466 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.282436    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-ac1065\""
Nov 02 21:03:41.282691 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.282541    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-ac1065-dockercfg-g7sf5\""
Nov 02 21:03:41.319527 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.319496    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbnl\" (UniqueName: \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.319720 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.319565    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.325174 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.325148    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:03:41.328981 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.328964    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.380384 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.380352    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:03:41.420680 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.420640    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.420907 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.420687    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjrf\" (UniqueName: \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.420907 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.420817    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.421057 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.420956    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbnl\" (UniqueName: \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.421430 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.421412    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.474528 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.474459    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbnl\" (UniqueName: \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\") pod \"payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.500533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.500499    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:41.521640 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.521606    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjrf\" (UniqueName: \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.521772 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.521699    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.522591 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.522566    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.576918 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.576886    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjrf\" (UniqueName: \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\") pod \"payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.643336 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.643300    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:41.865584 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.865556    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:03:41.867067 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:03:41.867044    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34dbfeb_a679_4a48_be23_1324ef0b284b.slice/crio-8bf956a94dc8296567038b6b67e36b652c4bec441dac1740f4bd6c2beb426b3d WatchSource:0}: Error finding container 8bf956a94dc8296567038b6b67e36b652c4bec441dac1740f4bd6c2beb426b3d: Status 404 returned error can't find the container with id 8bf956a94dc8296567038b6b67e36b652c4bec441dac1740f4bd6c2beb426b3d
Nov 02 21:03:41.910712 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:41.910684    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" event={"ID":"e34dbfeb-a679-4a48-be23-1324ef0b284b","Type":"ContainerStarted","Data":"8bf956a94dc8296567038b6b67e36b652c4bec441dac1740f4bd6c2beb426b3d"}
Nov 02 21:03:42.008839 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.008809    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:03:42.010469 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:03:42.010442    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603f8b45_8996_413d_aa6a_5fbe490b9186.slice/crio-7ab9105c91c8a0f7bb08b2fd2111aa294d06772f40b72ecb285b61ba4ead0abb WatchSource:0}: Error finding container 7ab9105c91c8a0f7bb08b2fd2111aa294d06772f40b72ecb285b61ba4ead0abb: Status 404 returned error can't find the container with id 7ab9105c91c8a0f7bb08b2fd2111aa294d06772f40b72ecb285b61ba4ead0abb
Nov 02 21:03:42.924620 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.924572    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" event={"ID":"e34dbfeb-a679-4a48-be23-1324ef0b284b","Type":"ContainerStarted","Data":"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"}
Nov 02 21:03:42.925135 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.924676    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:42.926217 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.926187    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" event={"ID":"603f8b45-8996-413d-aa6a-5fbe490b9186","Type":"ContainerStarted","Data":"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"}
Nov 02 21:03:42.926354 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.926236    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" event={"ID":"603f8b45-8996-413d-aa6a-5fbe490b9186","Type":"ContainerStarted","Data":"7ab9105c91c8a0f7bb08b2fd2111aa294d06772f40b72ecb285b61ba4ead0abb"}
Nov 02 21:03:42.926354 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:42.926293    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:43.032171 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.032114    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" podStartSLOduration=2.03209896 podStartE2EDuration="2.03209896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:43.030931464 +0000 UTC m=+4311.905202354" watchObservedRunningTime="2026-11-02 21:03:43.03209896 +0000 UTC m=+4311.906369850"
Nov 02 21:03:43.140439 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.140385    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" podStartSLOduration=2.140371537 podStartE2EDuration="2.140371537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:43.137464871 +0000 UTC m=+4312.011735760" watchObservedRunningTime="2026-11-02 21:03:43.140371537 +0000 UTC m=+4312.014642426"
Nov 02 21:03:43.394516 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.394482    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:03:43.398460 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.398443    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.441441 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.441408    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.441441 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.441444    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.441649 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.441462    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2rz\" (UniqueName: \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.441649 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.441631    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.448636 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.448610    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-ac1065-serving-cert\""
Nov 02 21:03:43.452136 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.452111    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:03:43.543283 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.543240    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.543498 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.543292    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.543498 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.543321    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2rz\" (UniqueName: \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.543498 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.543465    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.543498 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:43.543470    5099 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-serving-cert: secret "maas-api-e2e-authcfg-http-ac1065-serving-cert" not found
Nov 02 21:03:43.543725 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:43.543559    5099 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls podName:f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d nodeName:}" failed. No retries permitted until 2026-11-02 21:03:44.043537292 +0000 UTC m=+4312.917808160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls") pod "maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" (UID: "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d") : secret "maas-api-e2e-authcfg-http-ac1065-serving-cert" not found
Nov 02 21:03:43.544276 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.544252    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.546304 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.546280    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:43.597952 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:43.597925    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2rz\" (UniqueName: \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:44.058227 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:44.058191    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:44.060915 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:44.060886    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:44.312035 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:44.311947    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:44.677595 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:44.677569    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:03:44.680067 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:03:44.680033    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b7c66e_9e5b_4d6d_a6d4_f69a54b6314d.slice/crio-63a8f9e7a9f1e2a31302a58dbfb30c952438f546149655b0484e775deb6f1896 WatchSource:0}: Error finding container 63a8f9e7a9f1e2a31302a58dbfb30c952438f546149655b0484e775deb6f1896: Status 404 returned error can't find the container with id 63a8f9e7a9f1e2a31302a58dbfb30c952438f546149655b0484e775deb6f1896
Nov 02 21:03:44.938180 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:44.938092    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" event={"ID":"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d","Type":"ContainerStarted","Data":"63a8f9e7a9f1e2a31302a58dbfb30c952438f546149655b0484e775deb6f1896"}
Nov 02 21:03:45.944990 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:45.944952    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" event={"ID":"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d","Type":"ContainerStarted","Data":"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"}
Nov 02 21:03:45.945409 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:45.945145    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:46.054795 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:46.054726    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" podStartSLOduration=2.152315423 podStartE2EDuration="3.054711954s" totalImagesPullingTime="902.396531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:46.052279591 +0000 UTC m=+4314.926550504" watchObservedRunningTime="2026-11-02 21:03:46.054711954 +0000 UTC m=+4314.928982842"
Nov 02 21:03:46.093651 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:03:46.093614    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:03:51.956493 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:51.956463    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:03:53.934151 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:53.934114    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:03:53.934609 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:53.934356    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:03:56.858854 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:56.858824    5099 scope.go:122] "RemoveContainer" containerID="05bee947228d064c63a345e3fbcd57f38960369aac253547281013e5997237a7"
Nov 02 21:03:56.867851 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:03:56.867832    5099 scope.go:122] "RemoveContainer" containerID="6b69e2412e595681a5d32294f35ca99e5ec7d12abe9fd8c1417ef03d3cc133b3"
Nov 02 21:04:16.105324 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:04:16.105286    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:04:46.093626 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:04:46.093537    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:05:16.094683 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:05:16.094638    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:05:34.848841 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:34.848807    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:05:34.960279 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:34.960242    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:05:39.849727 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:39.849677    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" podUID="603f8b45-8996-413d-aa6a-5fbe490b9186" containerName="payload-processing" containerID="cri-o://8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec" gracePeriod=25
Nov 02 21:05:39.961471 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:39.961432    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" podUID="e34dbfeb-a679-4a48-be23-1324ef0b284b" containerName="payload-pre-processing" containerID="cri-o://20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51" gracePeriod=25
Nov 02 21:05:40.116912 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.116614    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:05:40.221076 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.221051    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:05:40.291435 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.291402    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\") pod \"603f8b45-8996-413d-aa6a-5fbe490b9186\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") "
Nov 02 21:05:40.291767 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.291530    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\" (UniqueName: \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\") pod \"603f8b45-8996-413d-aa6a-5fbe490b9186\" (UID: \"603f8b45-8996-413d-aa6a-5fbe490b9186\") "
Nov 02 21:05:40.291882 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.291773    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume" pod "603f8b45-8996-413d-aa6a-5fbe490b9186" (UID: "603f8b45-8996-413d-aa6a-5fbe490b9186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:05:40.291960 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.291945    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603f8b45-8996-413d-aa6a-5fbe490b9186-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:40.293665 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.293644    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf" pod "603f8b45-8996-413d-aa6a-5fbe490b9186" (UID: "603f8b45-8996-413d-aa6a-5fbe490b9186"). InnerVolumeSpecName "kube-api-access-qrjrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:05:40.392760 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.392670    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\" (UniqueName: \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\") pod \"e34dbfeb-a679-4a48-be23-1324ef0b284b\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") "
Nov 02 21:05:40.392938 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.392901    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\") pod \"e34dbfeb-a679-4a48-be23-1324ef0b284b\" (UID: \"e34dbfeb-a679-4a48-be23-1324ef0b284b\") "
Nov 02 21:05:40.393141 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.393116    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjrf\" (UniqueName: \"kubernetes.io/projected/603f8b45-8996-413d-aa6a-5fbe490b9186-kube-api-access-qrjrf\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:40.393275 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.393255    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume" pod "e34dbfeb-a679-4a48-be23-1324ef0b284b" (UID: "e34dbfeb-a679-4a48-be23-1324ef0b284b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:05:40.394754 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.394734    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl" pod "e34dbfeb-a679-4a48-be23-1324ef0b284b" (UID: "e34dbfeb-a679-4a48-be23-1324ef0b284b"). InnerVolumeSpecName "kube-api-access-hrbnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:05:40.494157 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.494123    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbnl\" (UniqueName: \"kubernetes.io/projected/e34dbfeb-a679-4a48-be23-1324ef0b284b-kube-api-access-hrbnl\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:40.494157 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.494152    5099 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34dbfeb-a679-4a48-be23-1324ef0b284b-plugins-config-volume\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:40.591773 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.591739    5099 generic.go:356] "Generic (PLEG): container finished" podID="e34dbfeb-a679-4a48-be23-1324ef0b284b" containerID="20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51" exitCode=0
Nov 02 21:05:40.591973 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.591815    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"
Nov 02 21:05:40.591973 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.591824    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" event={"ID":"e34dbfeb-a679-4a48-be23-1324ef0b284b","Type":"ContainerDied","Data":"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"}
Nov 02 21:05:40.591973 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.591880    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5" event={"ID":"e34dbfeb-a679-4a48-be23-1324ef0b284b","Type":"ContainerDied","Data":"8bf956a94dc8296567038b6b67e36b652c4bec441dac1740f4bd6c2beb426b3d"}
Nov 02 21:05:40.591973 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.591909    5099 scope.go:122] "RemoveContainer" containerID="20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"
Nov 02 21:05:40.593326 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.593298    5099 generic.go:356] "Generic (PLEG): container finished" podID="603f8b45-8996-413d-aa6a-5fbe490b9186" containerID="8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec" exitCode=0
Nov 02 21:05:40.593432 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.593349    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"
Nov 02 21:05:40.593432 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.593353    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" event={"ID":"603f8b45-8996-413d-aa6a-5fbe490b9186","Type":"ContainerDied","Data":"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"}
Nov 02 21:05:40.593432 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.593383    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg" event={"ID":"603f8b45-8996-413d-aa6a-5fbe490b9186","Type":"ContainerDied","Data":"7ab9105c91c8a0f7bb08b2fd2111aa294d06772f40b72ecb285b61ba4ead0abb"}
Nov 02 21:05:40.602170 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.602146    5099 scope.go:122] "RemoveContainer" containerID="20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"
Nov 02 21:05:40.602473 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:05:40.602454    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51\": container with ID starting with 20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51 not found: ID does not exist" containerID="20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"
Nov 02 21:05:40.602546 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.602481    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51"} err="failed to get container status \"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51\": rpc error: code = NotFound desc = could not find container \"20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51\": container with ID starting with 20d0464a0d170dbc853fc07ed9d537a5e0f6f0a39c704db310bdd61a7e39ad51 not found: ID does not exist"
Nov 02 21:05:40.602546 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.602497    5099 scope.go:122] "RemoveContainer" containerID="8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"
Nov 02 21:05:40.610718 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.610702    5099 scope.go:122] "RemoveContainer" containerID="8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"
Nov 02 21:05:40.611023 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:05:40.610998    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec\": container with ID starting with 8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec not found: ID does not exist" containerID="8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"
Nov 02 21:05:40.611092 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.611030    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec"} err="failed to get container status \"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec\": rpc error: code = NotFound desc = could not find container \"8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec\": container with ID starting with 8dc58ac796db04387eb1c088cbba8471dc38ff0e0e87c34e790cca6607593bec not found: ID does not exist"
Nov 02 21:05:40.747248 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.747159    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:05:40.750090 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.750064    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ac1065-76687b99dc-8ljc5"]
Nov 02 21:05:40.852285 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.852244    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:05:40.862993 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:40.858425    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ac1065-b89758867-2svzg"]
Nov 02 21:05:41.257758 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:41.257722    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603f8b45-8996-413d-aa6a-5fbe490b9186" path="/var/lib/kubelet/pods/603f8b45-8996-413d-aa6a-5fbe490b9186/volumes"
Nov 02 21:05:41.258165 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:41.258150    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34dbfeb-a679-4a48-be23-1324ef0b284b" path="/var/lib/kubelet/pods/e34dbfeb-a679-4a48-be23-1324ef0b284b/volumes"
Nov 02 21:05:46.094145 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:05:46.094103    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:05:47.152688 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.151544    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:05:47.152688 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.151912    5099 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" podUID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" containerName="maas-api" containerID="cri-o://cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6" gracePeriod=30
Nov 02 21:05:47.439277 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.439255    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:05:47.569360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.569327    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\" (UniqueName: \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\") pod \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") "
Nov 02 21:05:47.569614 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.569384    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") pod \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") "
Nov 02 21:05:47.569614 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.569459    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\") pod \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") "
Nov 02 21:05:47.569614 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.569569    5099 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\") pod \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\" (UID: \"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d\") "
Nov 02 21:05:47.570035 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.569982    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca" pod "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" (UID: "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 21:05:47.571759 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.571729    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls" pod "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" (UID: "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:05:47.571914 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.571889    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz" pod "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" (UID: "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d"). InnerVolumeSpecName "kube-api-access-8l2rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 21:05:47.571988 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.571968    5099 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs" pod "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" (UID: "f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 21:05:47.634175 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.634138    5099 generic.go:356] "Generic (PLEG): container finished" podID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" containerID="cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6" exitCode=0
Nov 02 21:05:47.634360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.634179    5099 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"
Nov 02 21:05:47.634360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.634219    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" event={"ID":"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d","Type":"ContainerDied","Data":"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"}
Nov 02 21:05:47.634360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.634271    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl" event={"ID":"f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d","Type":"ContainerDied","Data":"63a8f9e7a9f1e2a31302a58dbfb30c952438f546149655b0484e775deb6f1896"}
Nov 02 21:05:47.634360 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.634294    5099 scope.go:122] "RemoveContainer" containerID="cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"
Nov 02 21:05:47.644026 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.644010    5099 scope.go:122] "RemoveContainer" containerID="cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"
Nov 02 21:05:47.644322 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:05:47.644303    5099 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6\": container with ID starting with cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6 not found: ID does not exist" containerID="cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"
Nov 02 21:05:47.644377 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.644330    5099 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6"} err="failed to get container status \"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6\": rpc error: code = NotFound desc = could not find container \"cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6\": container with ID starting with cc5f700714bd85f0576965eb708d4d3a65b6b32307d0fee8bf0465c7d01e5fe6 not found: ID does not exist"
Nov 02 21:05:47.683480 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.683381    5099 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-openshift-service-ca\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:47.683480 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.683446    5099 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2rz\" (UniqueName: \"kubernetes.io/projected/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-kube-api-access-8l2rz\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:47.683480 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.683466    5099 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-maas-api-tls\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:47.683480 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.683481    5099 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d-metrics-certs\") on node \"ip-10-0-133-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 21:05:47.793293 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.793253    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:05:47.795831 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:47.795804    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ac1065-5b6757d64-nsscl"]
Nov 02 21:05:49.256163 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:05:49.256128    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" path="/var/lib/kubelet/pods/f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d/volumes"
Nov 02 21:06:03.043264 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:03.043174    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"]
Nov 02 21:06:03.047952 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:03.047920    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-v8tqz"]
Nov 02 21:06:03.257006 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:03.256969    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb" path="/var/lib/kubelet/pods/2c84be02-27c7-4ca5-8c1b-a9e42cd9e0cb/volumes"
Nov 02 21:06:16.094557 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:06:16.094512    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:06:24.769944 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:24.769905    5099 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-vfwzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 21:06:24.770596 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:24.770539    5099 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-vfwzd" podUID="8143c2ae-2081-4d35-bd74-1a43b243de1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 21:06:30.042174 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:30.042130    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"]
Nov 02 21:06:30.089019 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:30.088982    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-85h4b"]
Nov 02 21:06:31.256723 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:31.256686    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cf4eb4-04f1-414c-b04c-a722b405715a" path="/var/lib/kubelet/pods/e6cf4eb4-04f1-414c-b04c-a722b405715a/volumes"
Nov 02 21:06:46.093477 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:06:46.093433    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:06:52.163314 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:52.163282    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 21:06:52.204948 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:52.204924    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-s5vct_15ebbd01-9145-45e5-965d-bcffc2f28d24/console-operator/2.log"
Nov 02 21:06:59.042192 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:59.042156    5099 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"]
Nov 02 21:06:59.089347 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:59.089317    5099 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b1d91f-p2srs"]
Nov 02 21:06:59.256456 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:06:59.256420    5099 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff4f8ee-9218-4e68-9112-152219e17004" path="/var/lib/kubelet/pods/eff4f8ee-9218-4e68-9112-152219e17004/volumes"
Nov 02 21:07:16.267427 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:07:16.267382    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:07:43.978940 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:43.978851    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-696cfc768f-hhwjb_e3843dff-d849-4187-a2ae-a65de3254375/manager/0.log"
Nov 02 21:07:44.739849 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:44.739816    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kbzxz_3bd3085e-d212-4c20-bdce-605011614404/manager/0.log"
Nov 02 21:07:46.092885 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:07:46.092846    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:07:47.448775 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:47.448744    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bc7c89d86-brd4t_2f631b71-7320-4b80-8f07-b0340438ac2e/maas-api/0.log"
Nov 02 21:07:48.390188 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:48.390156    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qvnk8_14e3d047-8c95-4241-b899-20a8466fd914/postgres/0.log"
Nov 02 21:07:50.546719 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:50.546687    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xmk4p_c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8/manager/0.log"
Nov 02 21:07:51.027421 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:51.027390    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lldds_ed5c8ea0-5f41-4610-bc9e-57e9d389ca10/manager/0.log"
Nov 02 21:07:51.575816 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:51.575765    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gwdj_bb3d21d6-d541-4a18-bf3f-339d2d830191/manager/0.log"
Nov 02 21:07:52.308595 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:52.308560    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789hrswz_b3a55bd0-1b80-46f9-bfc4-fea42d0895a4/istio-proxy/0.log"
Nov 02 21:07:52.537936 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:52.537907    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j8j94_70ddef60-e0ce-4a6a-924b-3224709e79d6/discovery/0.log"
Nov 02 21:07:54.109311 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:54.109283    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-jskmn_fda36b26-75e5-45af-8fa1-ec906d88185b/istio-proxy/0.log"
Nov 02 21:07:54.364562 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:54.364484    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-p27sn_fb1fe38d-88dd-4f15-8f9a-8613c5161003/istio-proxy/0.log"
Nov 02 21:07:54.565069 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:54.565037    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-vfwzd_8143c2ae-2081-4d35-bd74-1a43b243de1b/istio-proxy/0.log"
Nov 02 21:07:54.772621 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:54.772543    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-vgn28_d1fd74f9-e682-4a78-b870-00402c71d4f0/istio-proxy/0.log"
Nov 02 21:07:55.483045 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:55.483013    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv_70cd8afc-945c-45c0-bff2-09e51914337c/istio-proxy/0.log"
Nov 02 21:07:55.677642 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:55.677611    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tm9g5_83ce931f-cff8-4e59-8c3e-387889e247c0/payload-pre-processing/0.log"
Nov 02 21:07:55.882996 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:55.882965    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:07:56.090061 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:56.090033    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb8948dc-fpq67_6a8bf33f-7796-49c0-a375-7932aa291af0/router/0.log"
Nov 02 21:07:56.934480 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:56.934447    5099 scope.go:122] "RemoveContainer" containerID="f33f2bf2e6807c4e17786ef0f52ce18c271c9d454a07985bc6ce9661e065b4ae"
Nov 02 21:07:56.948344 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:56.948321    5099 scope.go:122] "RemoveContainer" containerID="8665628c56eb8e620ad1ca8eea0569348a2cd824d076cc0d139ad381de902b2f"
Nov 02 21:07:56.956748 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:56.956728    5099 scope.go:122] "RemoveContainer" containerID="cff9d65f355428cf94d420f7b9e7e3ae43a2edb9d0dfb563bc3ba2bc970815f6"
Nov 02 21:07:57.387866 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:57.387836    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7bnfm_8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7/main/0.log"
Nov 02 21:07:57.850355 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:07:57.850279    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6_8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26/main/0.log"
Nov 02 21:08:16.093434 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:08:16.093395    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:08:18.404129 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:18.404102    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-696cfc768f-hhwjb_e3843dff-d849-4187-a2ae-a65de3254375/manager/0.log"
Nov 02 21:08:19.001872 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:19.001840    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kbzxz_3bd3085e-d212-4c20-bdce-605011614404/manager/0.log"
Nov 02 21:08:21.208649 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:21.208621    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bc7c89d86-brd4t_2f631b71-7320-4b80-8f07-b0340438ac2e/maas-api/0.log"
Nov 02 21:08:22.099824 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:22.099775    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qvnk8_14e3d047-8c95-4241-b899-20a8466fd914/postgres/0.log"
Nov 02 21:08:23.937450 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:23.937419    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xmk4p_c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8/manager/0.log"
Nov 02 21:08:24.365514 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:24.365490    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lldds_ed5c8ea0-5f41-4610-bc9e-57e9d389ca10/manager/0.log"
Nov 02 21:08:24.866507 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:24.866481    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gwdj_bb3d21d6-d541-4a18-bf3f-339d2d830191/manager/0.log"
Nov 02 21:08:25.432157 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:25.432132    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789hrswz_b3a55bd0-1b80-46f9-bfc4-fea42d0895a4/istio-proxy/0.log"
Nov 02 21:08:25.642524 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:25.642492    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j8j94_70ddef60-e0ce-4a6a-924b-3224709e79d6/discovery/0.log"
Nov 02 21:08:27.194418 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:27.194389    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-jskmn_fda36b26-75e5-45af-8fa1-ec906d88185b/istio-proxy/0.log"
Nov 02 21:08:27.444011 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:27.443985    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-p27sn_fb1fe38d-88dd-4f15-8f9a-8613c5161003/istio-proxy/0.log"
Nov 02 21:08:27.672969 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:27.672943    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-vfwzd_8143c2ae-2081-4d35-bd74-1a43b243de1b/istio-proxy/0.log"
Nov 02 21:08:27.874181 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:27.874155    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-vgn28_d1fd74f9-e682-4a78-b870-00402c71d4f0/istio-proxy/0.log"
Nov 02 21:08:28.540637 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:28.540606    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-648cc4b8c5-w7qzv_70cd8afc-945c-45c0-bff2-09e51914337c/istio-proxy/0.log"
Nov 02 21:08:28.737702 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:28.737669    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-tm9g5_83ce931f-cff8-4e59-8c3e-387889e247c0/payload-pre-processing/0.log"
Nov 02 21:08:28.952536 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:28.952507    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:08:29.157773 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:29.157732    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb8948dc-fpq67_6a8bf33f-7796-49c0-a375-7932aa291af0/router/0.log"
Nov 02 21:08:30.174601 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:30.174572    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7bnfm_8e4c9172-9ae9-48bd-a3b5-ecd59c5217c7/main/0.log"
Nov 02 21:08:30.685669 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:08:30.685633    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8vq8s6_8954ba67-1f7e-4ab0-b3cf-3c88b9da5f26/main/0.log"
Nov 02 21:08:46.094052 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:08:46.094014    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:09:16.093406 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:09:16.093328    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:09:41.752292 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:09:41.752261    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:09:41.959422 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:09:41.959391    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:09:46.093461 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:09:46.093426    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:10:16.094988 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:10:16.094952    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:10:30.068213 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:30.068137    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:10:30.281025 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:30.280998    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5fzvg_33c764f6-f4d1-4c8c-89b0-e7df0291baa9/payload-processing/0.log"
Nov 02 21:10:40.207452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:40.207414    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g5cr4_b64f0c97-c8f6-4e42-ac7d-32b5d6694968/global-pull-secret-syncer/0.log"
Nov 02 21:10:41.324963 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:41.324934    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kst75_e9711f2a-a824-4a90-afa0-9c2bd6661f6a/connection-checker/0.log"
Nov 02 21:10:41.711976 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:41.711944    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sk775_16088935-ac22-423a-b525-b5051055e3dc/connection-checker/0.log"
Nov 02 21:10:42.114465 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:42.114441    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wbhwp_ea63f5a8-8b6a-4cd4-9cca-f0b4e4473acc/connection-checker/0.log"
Nov 02 21:10:42.967379 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:42.967340    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-szw22_b22ccf89-c8b1-46c3-a6ae-feaecb3a180f/konnectivity-agent/0.log"
Nov 02 21:10:44.252930 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:44.252904    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-26.us-west-2.compute.internal_ee94c0e3fb310e2d7d2c04225780f5ac/haproxy/0.log"
Nov 02 21:10:46.093347 ip-10-0-133-26 kubenswrapper[5099]: E1102 21:10:46.093304    5099 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-66zt6" containerName="extractor"
Nov 02 21:10:49.969802 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.969757    5099 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"]
Nov 02 21:10:49.970434 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970411    5099 cpu_manager.go:414] "containerMap: removing container" podUID="603f8b45-8996-413d-aa6a-5fbe490b9186" containerName="payload-processing"
Nov 02 21:10:49.970533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970436    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603f8b45-8996-413d-aa6a-5fbe490b9186" containerName="payload-processing"
Nov 02 21:10:49.970533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970468    5099 cpu_manager.go:414] "containerMap: removing container" podUID="e34dbfeb-a679-4a48-be23-1324ef0b284b" containerName="payload-pre-processing"
Nov 02 21:10:49.970533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970478    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34dbfeb-a679-4a48-be23-1324ef0b284b" containerName="payload-pre-processing"
Nov 02 21:10:49.970533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970505    5099 cpu_manager.go:414] "containerMap: removing container" podUID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" containerName="maas-api"
Nov 02 21:10:49.970533 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970513    5099 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" containerName="maas-api"
Nov 02 21:10:49.970841 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970650    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="603f8b45-8996-413d-aa6a-5fbe490b9186" containerName="payload-processing"
Nov 02 21:10:49.970841 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970667    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34dbfeb-a679-4a48-be23-1324ef0b284b" containerName="payload-pre-processing"
Nov 02 21:10:49.970841 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.970681    5099 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b7c66e-9e5b-4d6d-a6d4-f69a54b6314d" containerName="maas-api"
Nov 02 21:10:49.974027 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:49.974005    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.028953 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.028922    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"]
Nov 02 21:10:50.115986 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.115944    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-proc\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.115986 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.115990    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nt9\" (UniqueName: \"kubernetes.io/projected/5e0f664b-0348-4628-8298-b2f91c1142c7-kube-api-access-88nt9\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.116275 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.116074    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-podres\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.116275 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.116167    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-lib-modules\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.116275 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.116210    5099 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-sys\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.116966 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.116951    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"kube-root-ca.crt\""
Nov 02 21:10:50.130922 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.130889    5099 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxnk7\"/\"default-dockercfg-m52f5\""
Nov 02 21:10:50.130922 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.130889    5099 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"openshift-service-ca.crt\""
Nov 02 21:10:50.217280 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217245    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-proc\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217294    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nt9\" (UniqueName: \"kubernetes.io/projected/5e0f664b-0348-4628-8298-b2f91c1142c7-kube-api-access-88nt9\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217335    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-podres\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217365    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-lib-modules\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217376    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-proc\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217452 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217402    5099 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-sys\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217639 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217457    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-podres\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217639 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217483    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-lib-modules\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.217639 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.217499    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0f664b-0348-4628-8298-b2f91c1142c7-sys\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.259697 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.259608    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xmk4p_c3e2f5df-3ae1-4612-9c1a-ec5fc10479f8/manager/0.log"
Nov 02 21:10:50.271180 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.271153    5099 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nt9\" (UniqueName: \"kubernetes.io/projected/5e0f664b-0348-4628-8298-b2f91c1142c7-kube-api-access-88nt9\") pod \"perf-node-gather-daemonset-xjxfr\" (UID: \"5e0f664b-0348-4628-8298-b2f91c1142c7\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.287221 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.287195    5099 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:50.654843 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:50.654815    5099 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"]
Nov 02 21:10:50.656891 ip-10-0-133-26 kubenswrapper[5099]: W1102 21:10:50.656859    5099 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5e0f664b_0348_4628_8298_b2f91c1142c7.slice/crio-19644c7b3551ced412a76039fc003969b12309f9f8253cbe54caaf6293c0e639 WatchSource:0}: Error finding container 19644c7b3551ced412a76039fc003969b12309f9f8253cbe54caaf6293c0e639: Status 404 returned error can't find the container with id 19644c7b3551ced412a76039fc003969b12309f9f8253cbe54caaf6293c0e639
Nov 02 21:10:51.092257 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.092172    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lldds_ed5c8ea0-5f41-4610-bc9e-57e9d389ca10/manager/0.log"
Nov 02 21:10:51.489888 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.489851    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr" event={"ID":"5e0f664b-0348-4628-8298-b2f91c1142c7","Type":"ContainerStarted","Data":"051964d5a54167d5d44fc4600ed0ad3c732f7d6054184d501ac0b5203622a843"}
Nov 02 21:10:51.490098 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.489906    5099 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr" event={"ID":"5e0f664b-0348-4628-8298-b2f91c1142c7","Type":"ContainerStarted","Data":"19644c7b3551ced412a76039fc003969b12309f9f8253cbe54caaf6293c0e639"}
Nov 02 21:10:51.490098 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.489992    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:10:51.597570 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.597515    5099 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr" podStartSLOduration=2.5974997010000003 podStartE2EDuration="2.597499701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:51.596974912 +0000 UTC m=+4740.471245800" watchObservedRunningTime="2026-11-02 21:10:51.597499701 +0000 UTC m=+4740.471770589"
Nov 02 21:10:51.920374 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:51.920342    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gwdj_bb3d21d6-d541-4a18-bf3f-339d2d830191/manager/0.log"
Nov 02 21:10:57.294994 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:57.294965    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-nfmz7_45233480-479f-4e15-b887-48d7eae1a759/cluster-monitoring-operator/0.log"
Nov 02 21:10:57.505881 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:10:57.505852    5099 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-xjxfr"
Nov 02 21:11:00.914558 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:00.914528    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8f7f_4a3273d3-b9ac-479c-962f-98a89ffef16a/node-exporter/0.log"
Nov 02 21:11:01.314947 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:01.314861    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8f7f_4a3273d3-b9ac-479c-962f-98a89ffef16a/kube-rbac-proxy/0.log"
Nov 02 21:11:01.716336 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:01.716309    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8f7f_4a3273d3-b9ac-479c-962f-98a89ffef16a/init-textfile/0.log"
Nov 02 21:11:08.199290 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:08.199261    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qp4hc_5937fd38-1cab-48bc-998d-d909080bad40/prometheus-operator-admission-webhook/0.log"
Nov 02 21:11:12.009114 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:12.009078    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-55fd485789hrswz_b3a55bd0-1b80-46f9-bfc4-fea42d0895a4/istio-proxy/0.log"
Nov 02 21:11:12.341959 ip-10-0-133-26 kubenswrapper[5099]: I1102 21:11:12.341868    5099 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-j8j94_70ddef60-e0ce-4a6a-924b-3224709e79d6/discovery/0.log"